Brookhaven Science Associates

Brookhaven National Laboratory

Independent Oversight Report

SA 02-02

Review of Self-Assessment Programs

of the

Environment, Safety, Health and Quality Directorate

September 11, 2002

Prepared by:

John Usher, Reviewer Date

> Independent Oversight Office

Approved by:

Robert McNair, Manager Date

Independent Oversight Office

Contents

1.0 Introduction [1](#introduction)

1.1 Purpose and Scope [1](#purpose-and-scope)

1.2 Review Process [1](#review-process)

1.3 Review Criteria [2](#review-criteria)

2.0 Program Summary [3](#program-summary)

2.1 Criterion 1: Approach to Self-Assessment
[3](#criterion-1-approach-to-self-assessment)

2.2 Criterion 2: Deployment of Self-Assessment
[7](#criterion-2-deployment-of-self-assessment)

2.3 Criterion 3: Results from Self-Assessment
[8](#criterion-3-results-from-self-assessment)

3.0 Conclusions [12](#conclusions)

3.1 Strengths [12](#strengths)

3.2 Areas for Improvement [13](#areas-for-improvement)

3.3 IAP Management System Programmatic Issues
[13](#iap-management-system-programmatic-issues)

Exhibits

Exhibit 1 - Documents Reviewed [15](#__RefHeading___Toc17701332)

Exhibit 2 -- Personnel Interviewed [17](#__RefHeading___Toc17701334)

Exhibit 3 - Documents Referenced [18](#__RefHeading___Toc17701335)

## Introduction

### Purpose and Scope

This report presents the results of Independent Oversight's review of
the Self-Assessment (SA) Programs of the Environment, Safety, Health and
Quality (ESH&Q) Directorate. As an element of the Integrated Assessment
Program (IAP), the Independent Oversight (IO) Office is chartered to
independently verify the effectiveness of Brookhaven National
Laboratory's Self-Assessment Program. To accomplish this, Independent
Oversight performs a review of Directorates or other organizations
reporting to the Laboratory Director in accordance with the requirements
and guidelines provided in the IAP Management System Description and the
Integrated Assessment Subject Area. Reviews are identified and scheduled
in the FY 2002 Program Plan: Review of Organizational Self-Assessment
Programs \[FY02 IO Review Plan\].

FY02 reviews focus on the organization's "approach" to self-assessment,
the status of "deployment" and the "use of results" to improve
performance. Specifically, IO will review how the organization is
planning to implement the self-assessment program, how the organization
conducts the self-assessment program, and how the results from
self-assessment are analyzed and used to improve performance. In the
course of this review, comparison of the current status of the ESH&Q SA
Programs to that reported in **Independent Oversight Report SA 00-01,
Evaluation of the Environment, Safety, Health, and Quality Directorate
Self-Assessment Programs \[IO Report SA 00-01\]** was used to measure
progress since the FY 2000 evaluation.

### Review Process

This review formally began on April 9, 2002, with an inbriefing
conducted with personnel from ESH&Q. This meeting served to introduce
the IO Reviewer, John Usher, who provided information on review
objectives, methodology, review criteria, and schedule. The inbriefing
also served to identify key personnel involved in the planning and
implementation of self-assessment.

The following data and collection methods were used during the review:

-   Document Reviews (Exhibit 1)

```{=html}
<!-- -->
```
-   Interviews (Exhibit 2)

```{=html}
<!-- -->
```
-   Documents Referenced (Exhibit 3)

Interviews were conducted using prepared questions tailored to address
the review criteria (see Section 1.3) in terms of key business
factors/performance objectives. Notes were prepared during each
interview to record the information gathered. Comments, concerns, and
follow‑up actions were also documented after each interview.

An outbriefing was conducted on August 27, 2002, with personnel from
ESH&Q. At this meeting, the IO Reviewer summarized results of the review
and submitted a draft report for review by ESH&Q. The meeting also gave
ESH&Q personnel the opportunity to provide feedback to the IO Reviewer
on the review process.

### Review Criteria 

The IO assessment process evaluates an organization's self-assessment
program against the criteria published in the **FY02 IO Review Plan**.

In FY02, BNL Critical Outcome Measure 3.1.1.1, Overall Evaluation of the
Laboratory Self-Assessment Program, was included in Appendix B of the
BNL Prime Contract. In concurrence with the process developed, and to be
conducted, jointly with DOE-BAO to evaluate BNL's Self-Assessment
Program, review criteria are organized under "approach/definition,
deployment/implementation, and improvement/results" as documented in the
**BNL Assessment Program Evaluation Guide: Process for Evaluation of
Integrated Assessment Program**. These criteria are:

1.  Approach

```{=html}
<!-- -->
```
1.  Organizational procedures address regulatory and/or SBMS drivers for
    assessment activities.

2.  The scopes of planned assessment activities are comprehensive, have
    a strong technical basis, and are balanced with work activities.

3.  The method for conducting key scheduled assessments is defined and
    is commensurate with type of assessment planned and performance
    information desired.

4.  Management and stakeholder involvement commensurate with their
    responsibilities is evident.

5.  Key supporting organizational processes (tracking systems, causal
    analysis, critiques, etc.) and tools are developed.

```{=html}
<!-- -->
```
2.  Deployment

 

1.  Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.

2.  Assessments are documented and communicated as planned.

3.  Assessment results are evaluated/analyzed to a degree commensurate
    with the type of assessment. Strengths and corrective/improvement
    actions are identified.

4.  Management and stakeholder involvement is evident.

```{=html}
<!-- -->
```
3.  Results

```{=html}
<!-- -->
```
1.  Corrective/improvement actions are prioritized and tracked to
    closure. Change control for action due dates is timely and clearly
    reflects consideration to balance priorities.

2.  Evidence of timely self-identification of issues exists. Significant
    issues are brought to the attention of management and disclosed to
    regulatory/oversight agencies in accordance with contractual
    obligations.

3.  Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations.

4.  Evidence exists that assessment activities have resulted in
    identification of opportunities for improvement and awareness of
    vulnerabilities. Connection as appropriate into
    strategic/institutional plans.

Further detailed information supplementing the review criteria is
provided in the **FY02 IO Review Plan**
(<http://www.io.bnl.gov/safy02.htm>).

## Program Summary

This review covers the ESH&Q Directorate comprising the Environmental
Services Division (ESD), the Radiological Control Division (RCD), the
Safety and Health Services Division (SHSD), the Occupational Medicine
Clinic (OMC), the Training and Qualifications (T&Q) Office, and the
Quality Programs and Services Office (QP&SO). The Independent Oversight
Office was not included in this review.

A discussion and analysis of data collected for the review of the
Directorate is presented for each criterion. Areas that demonstrated
positive performance or programmatic strengths (criteria are exceeded)
are identified with a (+). Areas where criteria are met but not exceeded
are unmarked. Areas that indicated opportunities for improvement
(criteria not met or partially met) are noted using a (-).

### Criterion 1: Approach to Self-Assessment

The ESH&Q organizations listed above each prepared SA Plans for FY02.
Additionally, an ESH&Q Directorate Self-Assessment Program description
was prepared defining "a self-assessment program in support of the ESH&Q
mission and business interests for those aspects of performance
applicable Directorate-wide." The ESH&Q Directorate SA Program
description was prepared subsequent to the development of Division and
Office SA Plans, and there was no intention that the SA Plans explicitly
link to the Directorate-level SA Program. SA Plans for SHSD and QP&SO
are dated from 2001. The OMC Plan is based on a calendar-year cycle and
was prepared in March 2002. The remaining plans are dated from January
2002 through May 2002.

Ample evidence exists that organizational managers appropriately
consider regulatory and contractual drivers in developing performance
goals/objectives and in planning and scheduling assessment activities.
Scopes of organizational SA programs are comprehensive in covering all
of each organization's operational and business functions. Managers
choose appropriate methodologies for assessment activities. It is
evident that activities are appropriately prioritized. Managers and key
stakeholders, including DOE-BAO, are appropriately involved in planning
assessment activities. Key processes for supporting the organizational
SA programs are in place.

1.  **Organizational procedures address regulatory and/or SBMS drivers
    for assessment activities.**

Organizational SA Plans reference Critical Outcomes, Supporting
Assessment Measures (SAMs), and specifically identified "compliance"
areas. Some plans also refer to a historical artifact known as Other
Contractual Expectations (OCEs) to indicate linkage that previously
existed to the BNL Prime Contract. Some explicit linkages to Critical
Outcomes and Performance Objectives are outdated by virtue of the late
issuance of Appendix B to the Prime Contract in February 2002. The ESH&Q
Directorate SA description provides explicit linkage to the IAP
framework and lists relevant Critical Outcome measures and SAMs. ESD and
SHSD plans use a tabular format to clearly document linkage among
drivers (regulations, DOE orders, Prime Contract including SAMs, and
SBMS Management Systems and Subject Areas), performance objectives and
assessment activities. The T&Q plan attaches "Activities Detail" that
delineates organizational objectives. The QP&SO plan attaches the QP&SO
manager's performance goals thereby explicitly linking organizational
objectives to the manager's performance. The RCD SA Plan does not
explicitly list organizational objectives but links organizational
performance expectations to the Radiological Control Management System,
Triennial Assessment Program, "Required Self-Assessments" with attendant
specific performance expectations, and "example" indicators listed in
Section 7.2. The OMC plan provides explicit linkage to the IAP framework
and incorporates objectives and assessment activities accordingly;
Section 6.0 explicitly provides linkage to DOE Order 440.1A - Worker
Protection Management for DOE Contractor Employees, the Code of Ethical
Conduct of the American College of Occupational and Environmental
Medicine (ACOEM), and the standards of the Accreditation Association for
Ambulatory Health Care (AAAHC).

All applicable required assessments are identified in ESH&Q SA plans.
The ESH&Q Directorate plan includes assessments such as Tier I Safety
Inspections, Lockout/Tagout, and Work Controls. As noted previously,
other ESH&Q organizational plans incorporate these and other required
assessments as compliance-based assessments for BNL, DOE, and external
regulators. RCD explicitly incorporates the Triennial Assessment
Program. QP&SO includes the annual review of the Quality Assurance
Program (QAP) Description. OMC specifically incorporates the AAAHC
accreditation as well as compliance with DOE O 440.1A.

2.  **(+) The scopes of planned assessment activities are comprehensive,
    have a strong technical basis, and are balanced with work
    activities.**

The comprehensive nature of ESH&Q's SA Plans is demonstrated based on
organizational usage of the IAP (Baldrige) framework, the Critical
Outcome structure, and/or alternative constructs such as RCD's linkage
to the Triennial Assessment Program. As mentioned previously, ESH&Q and
OMC SA Plans contain explicit linkage to the IAP framework. T&Q and
QP&SO are linked to appropriate SBMS Management System Descriptions. ESD
and SHSD plans attach comprehensive, tabulated listings of measures
addressing all business processes, financial controls, human resources,
and customer satisfaction with products and/or services provided by each
organization.

SA Plans explicitly state that measures and assessment activities are
prioritized based on results of previous assessments, business and/or
operational risk, and available resources. RCD and SHSD plans explicitly
document priority levels. The OMC plan documents the use of the Quality
Improvement and Safety Committee (QISC) to establish priorities. ESD and
SHSD plans provide estimates of resources (staff hours) required for
assessment activities.

The effectiveness of previously completed corrective actions is
appropriately considered during planning. Effectiveness of implemented
actions is evaluated during follow-up or periodically scheduled
assessment activities. The SHSD plan states the need to assess "areas
where corrective actions had previously been implemented." The OMC plan
specifically incorporates follow-up activities by the QISC.
Organizations use customer feedback as a key measure of effectiveness of
actions and as a driver for additional assessment, analysis, and/or
action.

Year-end reports contain explicit mention of results of external
assessments and SA Plans evidence linkage to these results.
Organizations incorporate results from BAO assessments and other
external assessments by DOE. ESD specifically includes the results of,
and schedule for, the ISO 14001 registration audit in the SA Plan. OMC
includes the scheduled accreditation by the AAAHC. Lessons learned are
incorporated into SA Plans in this fashion, and lessons learned are also
developed through organizational involvement in peer groups, working
groups, associations, and subscriptions to newsletters and/or list
servers. The T&Q Office, for example, conducts benchmarking for
improvement using the American Society for Training and Development
(ASTD).

ESH&Q managers use approaches, scopes and schedules that minimize the
impact to work activities at BNL in the conduct of "horizontal" or
programmatic assessments. Managers for ESD, RCD, and SHSD meet with
their DOE-BAO counterparts to discuss and agree upon scopes and
schedules for assessments that minimize duplication and/or overlap and
impact upon BNL organizations being assessed.

3.  **(+) The method for conducting key scheduled assessments is defined
    and is commensurate with type of assessment planned and performance
    information desired.**

Organizational SA Plans contain tables and/or descriptions of approaches
used for specific assessment and other activities. These tables outline
methodologies used to measure specific performance. Detailed assessment
plans are prepared for programmatic assessments, such as SAMs, and other
key assessment activities including compliance-based assessments. QP&SO
developed plans for Management System Evaluations. Many assessment
activities conducted by ESD, RCD, and SHSD are routinely proceduralized
either in SBMS or internally to the organization obviating the need for
a separate assessment plan. In addition to audits/assessments, SA
programs incorporate other methods such as ongoing monitoring of
performance indicators, tracking of budgets and milestones, management
walkthroughs, customer surveys, document reviews, data sampling, and
performance observations. Two SA Plans explicitly include mention of an
increased emphasis on performance effectiveness. SHSD's plan states,
"assessments are designed to place more emphasis on driving behavior ...
as compared to previous assessments, which focused more towards program
development and implementation." T&Q's plan states, "FY02 assessments
are designed to monitor the implementation and effectiveness of
programs."

Managers effectively communicate their expectations such that all
employees understand their roles and responsibilities for
self-assessment including program coordination and/or support. Notably,
ESD, RCD, SHSD, and T&Q plans explicitly describe the responsibilities
of Self-Assessment Coordinators. Roles and responsibilities for
self-assessment for managers, staff, and SMEs are explicitly listed in
SA Plans while individuals' performance goals and/or R2A2s are also used
to communicate management's expectations. SA Plans and/or supplemental
records document ownership and schedules for assessments or other
activities.

Assessment activities are assigned to qualified staff including
appropriate use of peers and subject matter experts (SMEs). SMEs are
routinely involved in the conduct of Tier I and environmental
assessments. QP&SO facilitated the conduct of maturity evaluations of
the Training and Qualifications Management System and the Quality
Management System. The T&Q Office routinely invites SMEs to evaluate
both training course content and instructor's performance. The T&Q
Office has also offered their SMEs to participate in ESH&Q "in-field"
assessments wherein effectiveness of training may be an issue; ESH&Q
managers have yet to avail themselves of this offer. Managers ensure
that independent assessments and peer reviews are considered and
incorporated into the self-assessment process, as appropriate. Notable
examples include the ISO14001 registrar's audit, triennial audits of the
RadCon Program, SHSD's use of Liberty Mutual independent assessments,
QP&SO's independent assessment of the Quality Assurance Program, and
accreditation of OMC by AAAHC.

4.  **(+) Management and stakeholder involvement commensurate with their
    responsibilities is evident.**

Planning involves managers and staff members as appropriate to ensure
that all aspects of the organization's operations are evaluated. SA
plans are typically drafted by Self-Assessment Coordinators with input
from managers, reviewed by managers and staff, and then approved by the
Division or Office Manager. Goals/objectives are initially developed
with the involvement of responsible managers and staff and then
typically rolled up from basic group/section level to Division/Office
level. Organizations use periodic management team meetings to develop
and/or revise goals and objectives. OMC uses the QISC process to
establish and revise goals and objectives. SHSD conducts a Year-end
Review involving SHSD personnel as well as other stakeholders to
identify areas for improvement and to plan for the next fiscal year. SA
Plans were submitted to the BNL Deputy Director for Operations for his
review. The ESH&Q Directorate-level objectives and measures have been
reviewed and discussed at the biweekly ESH&Q Managers meeting. A call
for contractual performance objectives/measures for FY03 was recently
issued at the ESH&Q Managers meeting.

SA Plans evidence attention to customer requirements/expectations. ESH&Q
managers interact with BNL, DOE-BAO and other stakeholders to obtain
their input on performance expectations as well as feedback on
performance. ESH&Q SAM owners involve their DOE-BAO counterparts in
planning SAMs and continue to interact with them during the conduct of
the SAM activity. Additionally, OMC and T&Q regularly survey customers
for feedback regarding services provided.

5.  **(+) Key supporting organizational processes (tracking systems,
    causal analysis, critiques, etc.) and tools are developed.**

ESH&Q organizations have effective processes in place for analyzing,
tracking and trending the results of assessment activities. Among
mechanisms for implementing these supporting processes are periodic
staff/management meetings including the ESH&Q Managers Biweekly. SA
status and results are regularly on the agendas at many of these
meetings including the proposed quarterly performance measure reviews
with DOE-BAO. These meetings result in the identification and assignment
of actions designed to improve performance in ESH&Q organizations.
Additionally, specific action plans are prepared for some activities
such as programmatic assessments and SAMs. Actions are typically
recorded in minutes and tracked using a variety of tracking systems,
ranging from logbooks or Day Planners, to MS Outlook or other
computerized databases/spreadsheets (some on individual computers and
others on shared drives), to the Family Assessment Tracking System
(FATS). For example, the ESH&Q Directorate, ESD, QP&SO, RCD, and SHSD
use FATS to track actions. It was noted during interviews that
Self-Assessment Coordinators in ESD and SHSD did not have ready access
to their respective FATS through interfaces on their "Family Reporting"
screens. Key Laboratory-level actions/commitments are formally tracked
in the Institutional ATS or CCTS.

It is noted that more formalized analysis of events and conditions to
determine causes is occurring within ESH&Q. A Subject Area is being
developed to institutionalize this process at BNL. Documentation of
these analyses is evident for Occurrence Reports, PAAA noncompliances,
and some internal and external assessments. QP&SO has facilitated
analyses within ESH&Q as well as for other BNL organizations. There are
also trending activities underway throughout ESH&Q. Examples include
results of Tier I inspections and injury reports. RCD prepares a monthly
Facility Support Services report incorporating issues identified around
the Laboratory. Information from Occurrence Reports, Radiological
Awareness Reports (RARs), and Nonconformance Reports (NCRs) is entered
into a database to establish a basis for trending analyses. The ORPS
Coordinator and the PAAA Coordinator periodically analyze Occurrence
Reports and PAAA noncompliances, respectively, to identify trends in
performance and/or programmatic issues across BNL.

SA Coordinators and/or organizational administrators maintain records
that demonstrate effective planning, timely completion of activities,
and analysis of results. SA Plans are maintained as controlled
documents. Results of assessment activities are documented with reports
submitted to responsible managers. Some ESH&Q organizations (RCD, ESD,
and SHSD) have developed report templates that serve as useful tools
both for reporting and as checklists to ensure comprehensive
assessments. SBMS tools supplement this process. As noted previously,
formal action plans are prepared for some assessments; these plans are
reviewed and/or approved by stakeholders. Updates on action plans are
discussed at management meetings and documented in meeting notes or
minutes. The SA Coordinator for ESD documents the status of SA
activities on a quarterly basis and provides this information to ESD
management.

Available information systems, such as FATS and organizational websites,
are used effectively to support assessment activities. Shared drives
and/or webpages are used to communicate SA Plans, organizational
mission, goals and objectives, schedules for activities, and to track
some actions. ESH&Q organizations use webpages to enable communication
and to facilitate interactions with customers by providing ready sources
of services and products available. Organizational procedures, some with
attached SA tools, are available on websites.

### Criterion 2: Deployment of Self-Assessment

Assessment activities are conducted as scheduled. Schedules are adjusted
based on changing priorities as determined by organizational managers.
Results of assessment activities are documented and communicated to
responsible ESH&Q managers for review and analysis. Key assessment
results are regularly discussed and analyzed at management and/or
organizational meetings. DOE-BAO managers and SAM counterparts are
informed of assessment status and results at meetings with ESH&Q
managers and/or staff. Quarterly meetings with DOE-BAO, specifically to
discuss performance measures, are proposed. Managers evaluate assessment
results to identify both strengths and opportunities for improvement.
Necessary improvement actions are identified and communicated to
responsible managers and stakeholders.

1.  **Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.**

Examples of assessment reports from FY01 and FY02 and information
tracked using FATS evidence completion of most assessments per schedule.
Some OMC activities have been delayed due to uncertainties regarding
management of the organization, but OMC now appears to be back on track.
Periodic monitoring activities are being completed at the appropriate
monitoring frequency. The SAM process is driving many of the activities
for FY02, and on-schedule performance is reflected in the high scores
obtained for SAMs. ESH&Q organizations demonstrate flexibility in
response to changing circumstances and priorities. SA Plans are revised
annually; no FY02 SA Plans have been revised since initial development
and approval.

2.  **(+) Assessments are documented and communicated as planned.**

Several examples of completed reports such as Management System
Evaluations, SAMs and other internal assessments were reviewed and are
referenced in Exhibit 1. RCD makes extensive use of report templates for
documenting results of "required" self-assessments. ESD and SHSD have
also developed and use templates to some extent. Responsible managers
produced copies of these reports and demonstrated familiarity with the
documented results. Documented results of SAMs are communicated as
planned to DOE-BAO. The assessment schedule and communication of results
are timed, in large part, around periodic Directorate and/or
organizational-level meetings to support timely management analyses and
decisions.

3.  **Assessment results are evaluated/analyzed to a degree commensurate
    with the type of assessment. Strengths and corrective/improvement
    actions are identified.**

ESH&Q organizations have effective processes in place to identify
opportunities for improvement. As mentioned previously, status and
results from assessment activities are regular agenda items at periodic
management and organizational meetings. Results from external
assessments are included in these meetings. Results of
discussions/analyses, identification of areas for improvement, and
planned actions are documented in meeting minutes. Improvement
opportunities are documented in FY01 organizational self-evaluation
reports and are input to FY02 Self-Assessment Plans. Reports of SAM
activities specifically identify areas for improvement and include
action plans. OMC explicitly includes these items on each agenda for
QISC meetings. T&Q management incorporates discussion of areas for
improvement and planned actions into T&Q Steering Committee meetings.
ESH&Q managers provide presentations to other BNL managers through
vehicles such as Management Council or Operations Council meetings.

4.  **Management and stakeholder involvement is evident.**

Organizational managers participate in the conduct of some assessment
activities. As per the SAM process, DOE-BAO counterparts are also
involved in the planning and conduct/observation of assessment
activities. Level 2 and Level 3 ESH&Q managers are owners of specific
assessment activities as indicated in SA Plans. ESH&Q managers conduct
some walkthroughs and worksite visits though not necessarily
formalized/documented. Efforts are underway to enhance the involvement
of ESH&Q managers in targeted walkthroughs. QP&SO and T&Q Office
managers led Management System Evaluations for the Quality Management
System and the Training and Qualifications Management System,
respectively. ESH&Q managers review the results of BNL customer surveys
and/or other feedback or interact/meet with BNL customers periodically
or on a less formally scheduled basis. ESH&Q managers also meet with
their DOE-BAO counterparts to provide updates on status and results of
assessment activities. ESD also meets periodically with regulatory
(federal, state, and local) stakeholders.

There is an evident customer focus throughout the ESH&Q Directorate. The
ESH&Q SA Plan includes a Directorate-level customer survey for FY02.
This survey may replace or supplement surveys planned and/or conducted
by ESH&Q organizations. OMC regularly solicits feedback from customers
using the Persons Receiving Care at the OMC (PRCOM) form. Comments are
reviewed and acted upon by OMC management. T&Q surveys attendees at
training courses. ESD, SHSD, and QP&SO all have planned customer surveys
for FY02.

### Criterion 3: Results from Self-Assessment

Most ESH&Q organizations prepared a Year-End Self-Evaluation Report
summarizing performance results for FY01 and identifying opportunities
for improvement in FY02. As noted previously, OMC's documents are based
on calendar year rather than fiscal year. These reports were not rolled
into a Directorate-level report for FY01. Critical Outcome performance
reporting was accomplished via computerized submittals to the BNL
Integrated Information Management System (IIMS).

Formal action plans are developed for selected assessment activities.
Actions are appropriately prioritized and tracked to closure using FATS
in most cases. FATS administrators and/or other responsible personnel
review the need for changes in action due dates and approve
rescheduling. Assessment reports reviewed provide evidence of timely
self-identification of issues/concerns. Results are reviewed at
management meetings and appropriately communicated to senior BNL
management and DOE-BAO. Examples of improved operational performance
were noted in ESH&Q. It is clear that organizational managers are aware
of vulnerabilities and areas for improvement. The ESH&Q Directorate has
not developed a Strategic Plan.

1.  **Corrective/improvement actions are prioritized and tracked to
    closure. Change control for action due dates is timely and clearly
    reflects consideration to balance priorities.**

Improvements are prioritized in accordance with business and operational
risks as evidenced by a key focus on Critical Outcomes, SAMs, other
contractual expectations, compliance with external regulations, and
internal measures, as documented largely in attachments to ESH&Q SA
Plans. RCD's SA Plan explicitly includes priorities for imrpovements
including as highest priority "action required for continued
organization mission or to eliminate a significant environmental,
health, or safety concern, or to restore compliance." As another
example, OMC's priorities are reflected in the assignment of a Quality
Management Improvement Program (QMIP) number to facilitate tracking by
the QISC.

In most ESH&Q organizations, higher priority (and other) actions are
tracked to closure using FATS. SA Plans reference ESH Standard 1.2.1,
Corrective Action Management and Tracking for External and Internal
Assessments, and state that actions are developed and managed
accordingly. ESH&Q organizations are seen to manage actions largely
according to ESH Standard 1.2.1, though this process continues to
mature. For example, documented causal analysis is evident for DOE-BAO
assessments and ORPS or PAAA reportable events. Again, this is aligned
with priorities discussed previously. Extensions of action due dates are
approved by organizational managers and/or assessment owners in concert
with FATS administrators, who have been delegated this responsibility in
some cases. This is largely in conformance with ESH Standard 1.2.1 with
some deviation.

ESH&Q managers verify that actions are implemented as articulated in SA
Plans. OMC's SA Plan reflects a focus "on the implementation of
operational improvements" explicitly through follow-up by the QISC
including any follow-up assessment deemed appropriate by the OMC
manager. One stated purpose of the T&Q SA process is to "ensure
implementation" of actions with the SA coordinator explicitly charged to
track implementation. In RCD's SA Plan, the Division Manager has the
ultimate responsibility to ensure that actions are implemented with
Section Managers and the Quality Representative also ensuring
implementation. Other SA Plans identify similar responsibilities with
all organizations reviewing completed actions at management meetings.

2.  **Evidence of timely self-identification of issues exists.
    Significant issues are brought to the attention of management and
    disclosed to regulatory/oversight agencies in accordance with
    contractual obligations.**

Results of assessment activities are analyzed to identify potential
noncompliances with the Nuclear Safety Rules (Price-Anderson Amendments
Act - PAAA). Plans explicitly include the responsibility to communicate
findings to the PAAA Coordinator, although some plans mistakenly stated
that this communication should be to the PAAA Working Group. ORPS, RARs,
and NCRs are routinely forwarded to the PAAA Coordinator for review
along with reports of DOE-BAO assessments, SAM activities, and
Management System Evaluations.

Results of assessment activities are also analyzed to determine
potential impacts on SBMS documents. SBMS documents have been
issued/revised or are in the process of being issued/revised as a result
of improvement opportunities identified from self-assessment (and other)
activities. ESD and SHSD explicitly assign responsibilities and specific
assessment activities for the review and revision of SBMS Subject Areas
in their FY02 SA Plans. RCD has implemented the Procedure Review
Committee explicitly charged to improve requirements-based documents.
QP&SO facilitated Management System Evaluations with the intent of
reviewing and improving SBMS Management Systems. QP&SO also annually
reviews the Quality Assurance Program Description and submits this
document for DOE-BAO review and approval.

3.  **Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations.**

Improvements in services/products provided by ESH&Q organizations are
evident. Some improvements have been validated by follow-up assessment
activities and some by feedback from customers. OMC developed (in
concert with stakeholders) an improved Job Assessment Form, implemented
a new database - Occupational Health Manager, instituted an improved
format for medical records, and (with T&Q) facilitated the entry of
health/safety protocol assignments onto the Brookhaven Training
Management System (BTMS). As discussed previously, QP&SO developed and
facilitated the MSE process, improved the format of Quality
Representative meetings, sponsored causal analysis and corrective action
management workshops, and facilitated causal analysis for BNL customers.
RCD developed report templates to improve the conduct of assessments by
Facility Support Representatives and Technicians and provided training
on the conduct of assessments, rotated staff assignments in order to
improve the consistency of Facility Support services across BNL,
encouraged improvement in staff qualifications, and improved training in
radiological protection. ESD was instrumental in retaining ISO 14001
registration for BNL's Environmental Management System, continues to
facilitate the Pollution Prevention process at BNL, and maintains the
ESD Monthly Highlights newsletter, a key communication tool within the
ESH&Q Directorate and BNL. SHSD implemented evident improvements in the
Chemical Management System, Industrial Hygiene field presence and
support, and the reporting and analysis of occupational injury/illness
and worker's compensation issues. T&Q developed several new web-based
and computer-based training courses, improved the utility, accuracy, and
scope of content on the BTMS and the Guest Information System, and
developed fitness training programs for BNL firefighters.

4.  **Evidence exists that assessment activities have resulted in
    identification of opportunities for improvement and awareness of
    vulnerabilities. Connection as appropriate into
    strategic/institutional plans.**

Improvement actions are mostly acted upon at the organizational or
Directorate level although some areas for improvement have been
communicated to the Laboratory level through documentation in year-end
organizational Self-Evaluation Reports or input to BNL's Annual
Self-Evaluation. As discussed above, results have been used to address
vulnerabilities and to improve SBMS Management Systems and Subject
Areas. The ESH&Q Directorate is planning to develop a Strategic Plan.
There is evident linkage from organizational goals and objectives to the
BNL Institutional Plan in specific areas such as Environmental
Management, Safety and Health, and Quality Management. There is,
however, little or no linkage evident between SA Plans to higher level
goals documented in the Institutional Plan under ESH&Q Management.

ESH&Q organizations demonstrated their responsiveness to other
independent or external assessments as exemplified via improvements
addressing the Areas for Improvement and the Recommendations made in
**IO Report SA 00-01**:

-   **Most Plans do not describe nor assign roles and responsibilities
    for operating and maintaining the organization's SA Program.**
    Organizational SA Plans for FY02 include clear descriptions of roles
    and responsibilities for self-assessment. SA Plans explicitly and
    clearly document the role of self-assessment coordinator and list a
    comprehensive set of responsibilities.

```{=html}
<!-- -->
```
-   **Most Plans do not describe processes for the analysis and/or
    evaluation of assessment results nor discuss the role of managers in
    these processes.** FY02 SA Plans include a comprehensive list of
    responsibilities for managers including the analysis and evaluation
    of results of assessment activities. FY02 SA Plans contain
    descriptions of analysis/evaluation processes used within each ESH&Q
    organization. The primary mechanisms used for analysis/evaluation
    are management teams and/or management meetings. SHSD conducts a
    year-end review consisting of presentations on performance,
    evaluation of results, and identification of areas for improvement
    and planned actions. Review attendees include SHSD personnel, ESH&Q
    Directorate managers, and other stakeholders.

```{=html}
<!-- -->
```
-   **Most Plans do not describe processes used to support the
    self-assessment process. Support responsibilities are not
    assigned.** Key supporting elements such as FATS are explicitly
    mentioned in FY02 SA Plans. In addition to responsibilities assigned
    to self-assessment coordinators, quality representatives and FATS
    administrators are explicitly assigned support roles for
    self-assessment in FY02 SA Plans, in individual performance goals
    and in R2A2s.

```{=html}
<!-- -->
```
-   **Most Plans do not describe formal reporting and/or documentation
    requirements. Most organizations do not manage Plans as Internal
    Controlled Documents.** FY02 SA Plans explicitly describe report
    requirements and processes. Some organizations incorporate report
    templates into their SA Plans. FY02 SA Plans are managed as
    internally controlled documents and are reviewed and approved by
    organizational managers.

```{=html}
<!-- -->
```
-   **Most Plans do not provide an estimate of resources required to
    conduct the organization's SA Program. Most Plans do not provide
    evident prioritization of assessment activities.** FY02 SA Plans
    explicitly include estimates of resources required for assessment
    activities or note that assessment activities are an integral part
    of the function of the organization and do not incorporate separate
    resource estimates. Some FY02 SA Plans describe specific
    prioritization methodologies for both assessment activities and
    planned actions. Most SA Plans differentiate priorities by
    identification as linked to Critical Outcomes, "required" and/or
    compliance-based, or internal assessments.

Results of assessment activities are analyzed to identify potential
lessons learned for communication to the BNL community and beyond. ESH&Q
SA Plans explicitly include the responsibility to communicate
appropriate information to the BNL Lessons Learned Coordinator for
review and publication. ESH&Q personnel have authored many BNL Lessons
Learned over the course of recent years. While difficult to measure
explicitly, it may be concluded that operational improvements and/or
reductions in vulnerabilities in environment, safety, and health areas
have resulted for these published lessons. Lessons learned are also
communicated via ES&H Coordinator meetings, Quality Representative
meetings, and Training Coordinator meetings, for example.

The draft Institutional Plan FY 2003 -- FY 2007, Section 7.4, ESH&Q
Management, describes a goal that includes operations that are
cost-effective. The section more explicitly states, "making every effort
to reduce indirect costs by eliminating redundancy and non-value added
work", and lists "new approaches to achieving efficient support." These
approaches include shared resources between support units, purchased
services from support organizations, purchased services from outside
BNL, and enhanced resource sharing and integration with corporate
partner and sister laboratories. While it is evident that organizational
managers are aware of, and manage their operations according to this
goal, there is no explicit linkage to these approaches in ESH&Q SA Plans
and few measures that directly address these approaches.

## Conclusions

ESH&Q organizations evidence considerable improvement in their
organizational Self-Assessment Programs since FY00. Programs are more
formally defined and more comprehensive in scope. Approaches to
self-assessment activities include more than just traditional audits and
represent a performance-based concept. Roles and responsibilities for
self-assessment are clearly defined. Processes for documenting results
of activities, analyzing results, identifying opportunities for
improvement, and tracking actions are explicitly described.
Self-assessment Plans document the closure of an annual cycle of
continuous improvement and represent a forward-looking approach, but the
lack of an ESH&Q Directorate Strategic (long-range) Plan and lack of
explicit linkage to the BNL Institutional Plan may hinder achievement of
higher-level goals and objectives.

### Strengths 

-   ESH&Q organizational SA Programs are comprehensive in scope
    encompassing operations/services, ES&H areas, business/financial
    aspects, and human resources. SA Plans contain clear descriptions of
    approaches for assessment activities appropriate for the type of
    performance information desired.

-   ESH&Q managers interviewed uniformly articulated commitment to
    improving organizational and individual performance as reflected in
    the services and products provided by ESH&Q organizations. Notably,
    managers indicated a desire for further improvement even in areas
    that were highly rated.

-   ESH&Q managers demonstrated considerable involvement in all
    dimensions of the SA process. Managers are clearly involved in
    establishment of performance objectives, planning of assessment
    activities, conduct of activities, analysis of assessment results,
    identification of opportunities for improvement and improvement
    actions, and tracking and follow-up on actions.

-   ESH&Q managers have clearly communicated expectations, roles and
    responsibilities for self-assessment. These are explicitly
    documented in organizational SA Plans, R2A2s, and/or individual
    performance goals. SA Plans include responsibilities of SA
    Coordinators.

-   Across the ESH&Q Directorate, there is considerable, evident
    customer focus. Organizations obtain customer feedback through
    surveys, focus groups and one-on-one interaction. Some organizations
    involve DOE-BAO in the scheduling and scoping of assessment
    activities.

-   ESH&Q organizations make extensive use of FATS for tracking actions
    resulting from assessment activities. SA Coordinators and/or FATS
    Administrators provide regular updates to managers on action status.
    Action status is an agenda item at organizational meetings.

-   ESH&Q organizations evidenced effective processes and tools as part
    of their SA Programs:

```{=html}
<!-- -->
```
-   OMC uses the Quality Improvement and Safety Committee to establish
    organizational objectives, plan assessment activities, analyze
    results of activities, identify areas for improvement, and follow up
    on planned actions through the Quality Management and Improvement
    Program;

-   The QA Representative for RCD has developed 14 report templates for
    required self-assessments and trains the Facility Support
    Representatives who conduct these assessments;

-   QP&SO has developed a process for conducting Management System
    Evaluations and facilitates these evaluations by Management System
    owners and points of contact;

-   T&Q regularly invites SMEs to observe and evaluate training courses
    and instructor performance to ensure currency of training content
    and effective delivery by instructors;

-   RCD publishes the Facilities Support Monthly Report incorporating
    organization-specific and Lab-wide issues and selected performance
    indicators;

-   The SA Coordinator for ESD prepares a Quarterly Performance Report
    documenting the status of assessment activities delineated in the SA
    Plan;

-   ESD communicates ESH related issues and project/action status by
    periodically publishing the ESD Highlights and distributing it to
    stakeholders in ESH&Q and elsewhere at BNL;

-   SHSD conducts a year-end review of organizational performance
    comprising presentations and discussion focusing on performance,
    strengths, areas for improvement, and planned actions.

### Areas for Improvement 

-   Some SA Plans were not completed by the end of the first quarter of
    FY01 (December 2001) as expected by Laboratory management (E-mail
    from T. Sheridan, Oct. 19, 2001).

> **Recommendation:** ESH&Q managers should ensure that they are aware
> of Laboratory expectations (See 3.3 below.) for completion of annual
> evaluations and revision of self-assessment programs. Managers should
> adhere to established expectations and schedules.

-   The RCD SA Plan does not include an explicit listing of
    organizational performance objectives.

> **Recommendation:** The RCD Manager should ensure that organizational
> performance objectives and measures are listed in the SA Plan.

-   ESH&Q organizations have not involved SMEs from the T&Q Office in
    assessments.

> **Recommendation:** ESH&Q managers should ensure that SMEs from T&Q
> are involved as appropriate in assessment activities wherein
> effectiveness of training may be an issue. In keeping with the
> Institutional Plan goal of efficient operations in ESH&Q, managers
> should ensure that resource sharing is considered for assessment
> activities.

-   ESD and SHSD SA Coordinators did not have ready access to their
    respective FATS Report pages. Both individuals were still linked to
    the Reactor Division FATS.

> **Recommendation:** FATS Administrators should ensure that
> organizational personnel have appropriate permissions and appropriate
> menu settings to enable full access to FATS.

-   ESH&Q has not developed a Strategic Plan.

> **Recommendation:** The Deputy Director for Operations should ensure
> that an ESH&Q Strategic Plan is developed as delineated under
> Objective 1.1 in the ESH&Q Directorate Self-Assessment Program for
> FY02.

### IAP Management System Programmatic Issues

It is acknowledged that the issues discussed below are known to BNL
management and that efforts are currently underway to address these
issues. These issues are noted again here for emphasis.

Some BNL managers (in ESH&Q and beyond) expressed confusion regarding
directions communicated by senior management regarding annual
evaluations and Self-Assessment Programs for FY02. It is recommended
that the Deputy Director for Operations ensure the review and revision
of the IAP Management System Description and the Integrated Assessment
Subject Area to explicitly document BNL's expectations for annual
evaluations and organizational self-assessment programs. These SBMS
documents should incorporate the annual cycle of self-assessment
including explicit dates for completion of key elements.

Some assessment owners were not aware of, or had not discharged, all
their responsibilities documented in ESH Standard 1.2.1, Corrective
Action Management and Tracking for External and Internal Assessments. It
is recommended that the Deputy Director for Operations ensure that BNL's
expectations for corrective action management are clearly documented and
communicated to assessment owners. BNL management should consider the
development of R2A2s for assessment owners and inclusion of these into
Level 1, Level 2, and Level 3 managers' R2A2s.

[]{#__RefHeading___Toc17701332 .anchor}Exhibit 1 - Documents Reviewed

1.  BNL ESH&Q Directorate Self-Assessment Program - FY 2002 (5/13/02)

2.  ESH&Q Bi-Weekly Managers Meeting Notes, April 23, 2002

3.  ESH&Q Bi-Weekly Managers Meeting Notes, May 21, 2002

4.  ESH&Q Objectives/Measures for review at 7/2/02 Managers Meeting
    (6/17/02)

5.  Memorandum dated April 22, 2002, D. Ports to File, "FY02 Independent
    Oversight Assessment Planning"

6.  BNL ESH&Q Directorate FY-01 Self-Evaluation Report (5/13/02)

7.  Agenda for ESH&Q All Hands Meeting -- June 13, 2002

8.  ESH&Q Directorate Self-Assessment Program, FY 2001, Revision March
    2001 (3/14/01)

9.  BNL Training & Qualifications Program Office Self-Assessment Plan -
    FY 02 (1/22/01)

10. Memorandum dated December 7, 2001, E. Tucker to B. Schwaner,
    "Self-Assessment Results for FY 2001"

11. Memorandum dated January 23, 2002, B. Schwaner to T. Sheridan,
    "Evaluation of BNL's Training and Qualifications Management System"

12. \[T&Q\] Management System Evaluation Question Set (12/5/01)

13. BNL Course Evaluation Form (09/2001)

14. BNL Instructor Evaluation Form (09/2001)

15. Committee Handbook -- Training and Qualifications Steering
    Committee, Effective Date: April 2001

16. BNL Safety and Health Services Division (SHSD) Self-Assessment Plan
    -- FY02 (12/28/01)

17. Safety and Health Services Division Self-Assessment \[Form\]

18. SHSD Organization Chart, April 2002

19. SHSD Self-Evaluation & Staff Recognition Meeting Agenda (9/20/01)

20. Travel-Related Worker Safety and Health \[Paper\]

21. Supporting Assessment Measure Approval Sheet -- Design Review Self
    Assessment (5/6/02)

22. Supporting Assessment Measure Approval Sheet -- NFPA 70E Electrical
    Safety Self-Assessment (5/9/02)

23. E-mail Memorandum dated June 5, 2002, R. J. Petricek to Level II
    Managers, "Upcoming FY02 Chemical Safety Performance Measures Random
    Surveys"

24. Memorandum dated March 18, 2002, O. White to Distribution, "SHSD
    Design Review Self Assessment -- Reviewer Response Rate Corrective
    Action Memo"

25. Design Review Status Table (6/7/02)

26. BNL Electrical Safety Program Self-Assessment FY02: Implementation
    of NFPA 70E (4/5/02)

27. Radiological Controls Division Self-Assessment Plan -- FY02, Rev. 0

28. Memorandum dated October 15, 2001, S. Layendecker to Radiological
    Control Division, "Fiscal Year 2001 Annual Self-Evaluation of the
    Radiological Control Division Performance"

29. Memorandum dated March 12, 2002, C. Schaefer to S. Layendecker,
    "Facilities Support January 2002 Monthly Report"

30. Memorandum dated April 11, 2002, C. Schaefer to S. Layendecker,
    "Facilities Support February 2002 Monthly Report"

31. Memorandum dated May 10, 2002, C. Schaefer to S. Layendecker,
    "Facilities Support March 2002 Monthly Report"

32. Memorandum dated January 9, 2002, S. Layendecker to Distribution,
    "Triennial Assessments"

33. Memorandum dated March 26, 2002, S. Layendecker to Distribution,
    "Transmittal of Triennial Assessment Report RCD TA 01 02"

34. Letter dated March 1, 2002, S. Mallette (BAO) to T. Sheridan (BNL),
    "Surveillance of Radiological Control Technician Oral Qualification
    Boards"

35. BNL Radiological News, March 2002

36. Letter dated April 12, 2002, T. Sheridan to F. Crescenzo (BAO),
    "Corrective Action Response to the Assessment of the Radiological
    Release Criteria Program at Brookhaven National Laboratory"

37. BNL ESD Self-Assessment Plan for FY 2002, January 2002

38. ESD FY 2002 Self-Assessment Plan, Quarterly Performance Report for
    2^nd^ Quarter FY02 (Draft -- 6/5/02)

39. Letter dated February 21, 2002, L. Cunniff to S. Mallette, "BNL
    Environmental Assessment of the Radiological and Non-Radiological
    Air Quality Programs"

40. Minutes for ESD Monthly Meeting, Thursday, June 27, 2002

41. Self-Assessment Plan - Occupational Medicine Clinic - CY 2002
    Initial, March 2002

42. Memorandum dated March 21, 2002, L. Sbarra to T. Sheridan, "CY 2001
    Self Assessment Year End Update"

43. Quality Programs & Services Office Self-Assessment Plan for FY2002

44. Quality Representatives Meeting Agenda (6/26/02)

45. Evaluation \[Form\] Quality Representatives Meeting -- June 26, 2002

46. ATS Action Detail Report for Action #134.2.10 -- Complete
    Verification of Quality Integration Project Implementation (6/7/02)

47. Memorandum dated May 13, 2002, R. Lebel to T. Sheridan, "Evaluation
    of BNL's Quality Management System"

48. Quality Management System Evaluation Question Set, Rev. 2, (3/29/02)

49. Letter dated May 15, 2002, T. Sheridan to S. Mallette (BAO),
    "Summary Report of BNL Quality Assurance Program Verification
    Process"

50. Letter dated July 10, 2002, F. Crescenzo (BAO) to T. Sheridan,
    "\[Approval of\] Brookhaven National Laboratory (BNL) Quality
    Assurance Program (QAP) Description"

[]{#__RefHeading___Toc17701334 .anchor}Exhibit 2 -- Personnel
Interviewed

1.  T. Sheridan, Deputy Director for Operations

2.  L. Cunniff, Manager, ESD

3.  L. Sbarra, Manager, OMC

4.  S. Layendecker, Manager, RCD

5.  O. White, Manager, SHSD

6.  B. Schwaner, Manager, T&Q Office

7.  R. Lebel, Manager, QP&S Office

8.  D. Ports, ESH&Q Directorate, IAP, PBM

9.  M. Davis, ESD, SA Coordinator

10. D. Maresca, OMC

11. M. Verderosa, RCD, QA Representative

12. R. Travis, SHSD, SA Coordinator

13. E. Tucker, T&Q, SA Coordinator

14. S. Stein, QPSO

15. T. Monahan, SHSD, SE Group Leader

16. C. Schaefer, RCD, FS Section Manager

17. B. Penn, ESH&Q Business Operations Manager

[]{#__RefHeading___Toc17701335 .anchor}Exhibit 3 - Documents Referenced

1.  SBMS Management System Description: Integrated Assessment Program,
    Issue Date: July 1999

2.  BNL Independent Oversight Office FY 2002 Program Plan: Review of
    Organizational Self-Assessment Programs (January 11, 2002)

3.  BNL Assessment Program Evaluation Guide: Process for Evaluation of
    Integrated Assessment Program (IAP), Critical Outcome Measure
    3.1.1.1, Overall Evaluation of the Laboratory Self-Assessment
    Program, February 12, 2002

4.  E-mail from T. Sheridan to Level I and Level Managers, Subject:
    Annual Self-Evaluation Report to DOE (October 19, 2001)

5.  BNL Issue and Decision Paper: Approval and Implementation of Subject
    Area "Integrated Assessment Program" (September 14, 2000)

6.  BNL Memorandum from Peter Paul to Department Chairs, Subject:
    Criteria for Semi-Annual Evaluation -- July 20, 2000

7.  U.S. Department of Energy Contract with Brookhaven Science
    Associates, DE-AC02-98CH10886

8.  Appendix B (to DE-AC02-98CH10886), Critical Outcomes, Objectives,
    and Performance Measures FY 2002 Brookhaven National Laboratory
    (Modification M077, May 31, 2002)

9.  Year End Self-Evaluation - Brookhaven National Laboratory - Fiscal
    Year 2001 (January 7, 2002)

10. Mid-Year Self-Evaluation - Brookhaven National Laboratory -- Fiscal
    Year 2002 (June 12, 2002)

11. Brookhaven National Laboratory Draft Institutional Plan FY 2003 --
    FY 2007, May 2002

12. SBMS Subject Area: Integrated Assessment, Effective Date: September
    2000

13. SBMS Subject Area: Internal Controlled Documents, Effective Date:
    August 2000

14. SBMS Subject Area: Performance Goals and Measures for Employees,
    Effective Date: October 2001

15. SBMS ESH Standard 1.2.1, Corrective Action Management and Tracking
    for External and Internal Assessments, Rev. 4, Effective Date:
    October 2001

16. Brookhaven National Laboratory Independent Oversight Report SA
    00-01, Evaluation of the Environment, Safety, Health, and Quality
    Directorate Self-Assessment Programs (February 29, 2000)