Brookhaven Science Associates

Brookhaven National Laboratory

Internal Audit and Oversight Report

SA 04-03

Review of the Self-Assessment Program

of the

Environment, Safety, Health and Quality Directorate

September 28, 2004

Prepared by:

John Usher, Reviewer Date

> Independent Oversight

Approved by:

Robert McNair, Manager Date

Independent Oversight

Contents

1.0 Introduction [1](#introduction)

1.1 Purpose and Scope [1](#purpose-and-scope)

1.2 Review Process [1](#review-process)

1.3 Review Criteria [2](#review-criteria)

2.0 Program Summary [3](#program-summary)

2.1 Criterion 1: Approach to Self-Assessment
[3](#criterion-1-approach-to-self-assessment)

2.2 Criterion 2: Deployment of Self-Assessment
[7](#criterion-2-deployment-of-self-assessment)

2.3 Criterion 3: Results from Self-Assessment
[11](#criterion-3-results-from-self-assessment)

3.0 Conclusions [13](#conclusions)

3.1 Strengths [13](#strengths)

3.2 Areas for Improvement [14](#areas-for-improvement)

3.3 IAP Management System Programmatic Issues
[15](#iap-management-system-programmatic-issues)

Exhibits

Exhibit 1 - Documents Reviewed [16](#__RefHeading___Toc68064789)

Exhibit 2 -- Personnel Interviewed [18](#__RefHeading___Toc68064790)

Exhibit 3 - Documents Referenced [19](#__RefHeading___Toc68064791)

## Introduction

### Purpose and Scope

This report presents the results of Independent Oversight's review of
the Self-Assessment (SA) Program of the Environment, Safety, Health and
Quality (ESH&Q) Directorate. As an element of the Integrated Assessment
Program (IAP), Independent Oversight (IO) is chartered to independently
verify the effectiveness of Brookhaven National Laboratory's
Self-Assessment Program. To accomplish this, IO personnel perform
reviews of Directorates or other organizations reporting to the
Laboratory Director in accordance with the requirements and guidelines
provided in the IAP Management System Description. Reviews are
identified and scheduled in the FY 2004 Program Plan: Review of
Organizational Self-Assessment Programs \[FY04 IO Review Plan\].

FY04 reviews focus on the organization's "approach" to self-assessment,
the status of "deployment," and the "use of results" to improve
performance. Specifically, IO will review how the organization is
planning to implement the self-assessment program, how the organization
conducts the self-assessment program, and how the results from
self-assessment are analyzed and used to improve performance. In the
course of this review, comparison of the current status of the ESH&Q SA
Program to that reported in **Independent Oversight Report SA 02-02,
Review of Self-Assessment Programs of the Environment, Safety, Health
and Quality Directorate \[IO Report SA 02-02\]** was performed to
measure progress since the FY 2002 evaluation.

### Review Process

This review formally began on June 25, 2004, with an inbriefing
conducted with personnel from ESH&Q. This meeting served to introduce
the IO Reviewer, John Usher, who provided information on review
objectives, methodology, review criteria, and schedule. The inbriefing
also served to inform the IO Reviewer about ESH&Q and to identify key
personnel involved in the planning and implementation of
self-assessment.

The following data and collection methods were used during the review:

-   Documents Reviewed (Exhibit 1)

```{=html}
<!-- -->
```
-   Interviews (Exhibit 2)

```{=html}
<!-- -->
```
-   Documents Referenced (Exhibit 3)

Interviews were conducted using prepared questions tailored to address
the review criteria (see Section 1.3) in terms of key business
factors/performance objectives. Notes were prepared during each
interview to record the information gathered. Comments, concerns, and
follow‑up actions were also documented after each interview.

An outbriefing was conducted on September 14, 2004, with personnel from
ESH&Q. At this meeting, the IO Reviewer summarized results of the review
and submitted a draft report for review by ESH&Q personnel. The meeting
also gave ESH&Q personnel the opportunity to provide feedback to the IO
Reviewer on the review process.

### Review Criteria 

The IO assessment process evaluates an organization's self-assessment
program against the criteria published in the **FY04 IO Review Plan**.
These criteria are:

1.  Approach

```{=html}
<!-- -->
```
1.  The scopes of planned assessment activities are comprehensive, have
    a strong technical basis, and are balanced with work activities.

2.  The methods for conducting key scheduled assessments are defined and
    are commensurate with types of assessments planned and performance
    information desired.

3.  Management and staff involvement commensurate with their
    responsibilities is evident.

4.  Key supporting organizational processes (tracking/trending systems,
    causal analysis, critiques, etc.) and tools are developed.

5.  Organizational plans/procedures address regulatory and/or SBMS
    drivers for assessment activities.

```{=html}
<!-- -->
```
2.  Deployment

 

1.  Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.

2.  Assessments are documented and communicated as planned.

3.  Assessment results are evaluated/analyzed to a degree commensurate
    with the type of assessment. Strengths are identified.
    Corrective/improvement actions are identified, prioritized, assigned
    to specific owners, and assigned due dates. All conditions are
    tracked to closure. Actions are validated as improving performance.

4.  Evidence of timely self-identification of issues exists. Significant
    issues are brought to the attention of management and disclosed to
    regulatory/oversight agencies in accordance with contractual
    obligations.

5.  Management involvement is evident.

```{=html}
<!-- -->
```
3.  Results

```{=html}
<!-- -->
```
1.  Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations, such as mission
    achievement and retention/expansion of core competencies.

2.  Evidence exists that there is an appropriate connection between
    results of organizational self-evaluation and development of
    strategic/institutional plans.

Further detailed information supplementing the review criteria is
provided in the **FY04 IO Review Plan**
(<http://www.io.bnl.gov/safy04.doc>).

## Program Summary

This review covers the ESH&Q Directorate comprising the Environmental
and Waste Management Services (EWMS) Division, Radiological Control
Division (RCD), Safety and Health Services Division (SHSD), Quality
Management Office (QMO), and Training and Qualifications (T&Q) Program
Office. Organizational/functional areas addressed in this review
included Business Operations Management. Each of these organizations,
with the exception of T&Q, has undergone major change during FY04: EWMS
resulted from the merger of the Environmental Services Division (ESD)
and the Waste Management Division (WMD), and a new manager for EWMS was
selected; RCD has a new manager; SHSD has an interim manager following
the departure of the previous manager; and QMO resulted from the
combination of the Quality Programs and Services Office (QPSO), the
Standards-Based Management System (SBMS) Office, and some functions
previously managed by the Office of Management Services (OMS), and a new
manager has been selected.

The Assistant Laboratory Director (ALD) for ESH&Q has stewardship
responsibilities for ten SBMS Management Systems: Environmental
Management System (EMS), Facility Safety, Hazardous Material
Transportation Safety, Integrated Assessment Program, Occupational
Safety and Health (OHSAS 18001), Quality Management, Radiological
Control, Standards-Based Management, Training and Qualifications, and
Worker Safety and Health.

A discussion and analysis of data collected for the review of the ESH&Q
Directorate is presented for each criterion. Areas that demonstrated
positive performance or programmatic strengths (criteria exceeded) are
identified with a **(+)**. Areas where criteria are met but not exceeded
are unmarked. Areas that indicated opportunities for improvement
(criteria not met or partially met) are noted using a **(-)**.

### Criterion 1: Approach to Self-Assessment

# The ESH&Q Directorate maintains an overall Directorate-level "Performance Management Program" document in addition to the Self-Assessment (SA) Plans being maintained by each of the Divisions and Offices listed previously. The Environmental, Safety, Health and Quality Directorate Performance Management Program 2004 (Revision 0 dated 6/3/04) was prepared by the QMO Manager and approved by the ALD. Environmental & Waste Management Services Division Self-Assessment Plan FY 2004 (Revision 0 dated 3/16/04) was approved by the EWMS Division Manager. Radiological Control Division Self-Assessment Plan, FY 04 (Revision 1 dated 6/02/04) was approved by the RCD Manager. Safety and Health Services Division Self-Assessment Plan FY 2004 (Revision 0F dated 5/21/04) was approved by the SHSD Interim Manager. Training and Qualifications Program Office FY04 Self-Assessment Plan (12/29/03, amended 6/29/04) was approved by the T&Q Program Office Manager. Quality Management Office Self-Assessment Plan Fiscal Year 2004 (Draft Revision 1 dated 7/01/04) was not yet approved by the QMO Manager. {#the-eshq-directorate-maintains-an-overall-directorate-level-performance-management-program-document-in-addition-to-the-self-assessment-sa-plans-being-maintained-by-each-of-the-divisions-and-offices-listed-previously.-the-environmental-safety-health-and-quality-directorate-performance-management-program-2004-revision-0-dated-6304-was-prepared-by-the-qmo-manager-and-approved-by-the-ald.-environmental-waste-management-services-division-self-assessment-plan-fy-2004-revision-0-dated-31604-was-approved-by-the-ewms-division-manager.-radiological-control-division-self-assessment-plan-fy-04-revision-1-dated-60204-was-approved-by-the-rcd-manager.-safety-and-health-services-division-self-assessment-plan-fy-2004-revision-0f-dated-52104-was-approved-by-the-shsd-interim-manager.-training-and-qualifications-program-office-fy04-self-assessment-plan-122903-amended-62904-was-approved-by-the-tq-program-office-manager.-quality-management-office-self-assessment-plan-fiscal-year-2004-draft-revision-1-dated-70104-was-not-yet-approved-by-the-qmo-manager. .unnumbered}

The scope of the SA program is largely comprehensive in addressing the
Directorate's operations, products, and services. Financial management
activities such as budget planning and ensuring adherence to spending
plans are evident as documented in SA Plans and Annual Budget
Presentations. Appropriate methodologies for assessment are described in
the SA Plans. Assessment activities are appropriately prioritized, some
explicitly. Managers, staff, and key stakeholders are appropriately
involved in planning assessment activities. Processes for supporting the
organizational SA programs are in place including use of Family
Assessment Tracking Systems (FATS). Evidence exists that organizational
managers appropriately consider SBMS, contractual, and regulatory
drivers in developing performance goals/objectives and in planning and
scheduling assessment activities, and FY04 BNL required assessments
relevant to ESH&Q organizations were acknowledged in some SA Plans.

1.  **The scopes of planned assessment activities are comprehensive,
    have a strong technical basis, and are balanced with work
    activities.**

The ESH&Q Directorate Performance Management Program (PMP) document
explicitly references Laboratory Strategic Imperatives, Directorate
Strategic Imperatives, Contractual Performance Measures, and "carry-over
issues from the FY03 BNL Self Evaluation and the DOE response to same."
The ESH&Q PMP document indicates that Division/Office SA Plans are
linked to Strategic Imperatives. The PMP (Section 4.0) states "The
results of the annual evaluation serve as a basis for improvement
actions and assessment activities for the subsequent year." SHSD and
EWMS SA Plans are clearly aligned with their respective "Division
Strategic Objectives." **(+)** SHSD's Plan explicitly includes
priorities for each assessment activity and states, "This ranking scheme
allows the most effective use of limited Division resources." RCD's SA
Plan incorporates RCD Strategic Imperatives. The RCD SA Plan also states
that assessment schedules are based on "budgetary constraints." The
draft QMO SA Plan specifically incorporates linkage to Contractual
Performance Measures and to Directorate Strategic Imperatives. The draft
QMO SA Plan was the only one issued after the ESH&Q SA Plan. **(-)** The
draft QMO SA Plan does not explicitly include the Laboratory Strategic
Imperative "Self-Assessment and Performance Measure Processes" as
indicated in ESH&Q PMP Section 2.3.4. **(-)** The draft QMO SA Plan does
not discuss the "projectizing" of SBMS listed as a Directorate Strategic
Imperative in Section 2.4 of the PMP.

The SHSD and EWMS SA Plan Attachments are specifically organized
according to the IAP (Baldrige) framework. The PMP is structured
according to Contractual Performance Measures, and Laboratory and
Directorate Strategic Imperatives. The RCD SA Plan acknowledges
structure from Triennial Assessments, Contractual Performance Measures,
and Division Strategic Imperatives. The T&Q SA Plan acknowledges
Laboratory Expectations (Part I) and Effectiveness and Continual
Improvement of T&Q Courses and Services (Part II). The QMO draft SA Plan
is structured according to Contractual Performance Measures and
Directorate Strategic Imperatives.

Most SA Plans are comprehensive in covering operational,
business/financial, customer satisfaction, and compliance. **(-)** RCD's
SA Plan does not include objectives other than compliance, while RCD's
FY03 Annual Self-Evaluation documents other performance measures
including specific expectations for Facility Support, Analytical
Services Laboratory, Instrumentation and Calibration, Personnel
Monitoring, Radiological Assistance Program, and Health Physics
Technical Support. RCD's SA Plan does not link to the previous year's
Self-Evaluation to address identified areas for improvement.

SA Plans evidence attention to DOE, other regulators, and BNL management
expectations and requirements through linkage with Contractual Measures,
compliance, and acknowledgment of SBMS. The ESH&Q PMP specifically
documents an area of "DOE unhappiness" with BNL performance under
"Carryover Issues from FY03." The PM Plan states that "these areas are
again (FY04) included in the Laboratory's contractual performance
measures." The RCD SA Plan specifically requires that the RCD Manager
"team with the \[BHSO\] counterpart in determining the scope of any
"collaborative" assessment scheduled for the upcoming fiscal year." The
RCD SA Plan also states that assessments schedules are based in part on
"customer/stakeholder views and expectations." SA Plans require that
internal customers be involved in planning. SA Coordinators are
typically required to interact with managers, subject matter experts
(SMEs), and staff in developing the SA Plan. The Laboratory T&Q Steering
Committee guides T&Q planning and operations.

The EWMS and SHSD SA Plans (Section 4.3) state, "The results of the
previous year's assessments and annual evaluation serve as a basis for
improvement actions and assessment activities for the subsequent year."
The T&Q SA Plan (Objective & Methodology) states, "\[FY04\] assessments
are designed to monitor the implementation and effectiveness of
programs, as compared to previous assessments, which focused more on the
development and implementation of new programs." There is no explicit
mention of consideration of effectiveness of previous corrective actions
during planning; however, interviewees acknowledged factoring this into
assessment planning, and some assessment activities include review of
previously implemented actions.

ESH&Q organizations incorporate scheduled external assessments
(especially including BHSO), audits, and inspections in their SA Plans.
The best example of this is the EMS Management Review which specifically
includes ISO 14001 Surveillance Audit, Regulatory Agency Compliance
Inspections, IA&O Independent Assessments, and DOE BHSO Assessments. T&Q
specifically included the results of the T&Q Review Team from the BNL
Operations Retreat in performance evaluation. RCD noted accreditations
and certifications by national and state organizations.

External lessons learned are incorporated into the self-assessment
process. The BNL Lessons Learned Program is managed by QMO. ESH&Q
personnel subscribe to DOE's Lessons Learned listserver. Personnel also
participate on external committees, are active members of
associations/societies, and attend conferences and workshops. EWMS and
SHSD personnel are afforded the opportunity "to attend one professional
development opportunity if funding is available." Some ESH&Q personnel
visit, and interact with personnel from, other DOE Laboratories. T&Q
uses American Society for Training and Development (ASTD) benchmarking
tools in evaluating performance.

2.  **The methods for conducting key scheduled assessments are defined
    and are commensurate with types of assessments planned and
    performance information desired.**

Assessment activities are assigned to qualified personnel including
appropriate use of subject matter experts. EWMS and SHSD SA Plans
(Section 2.3) explicitly include among "individuals responsible for
coordinating or conducting the assessment activities," subject matter
experts and Quality Assurance (QA) representatives. RCD self-assessments
and/or surveillances are assigned to Facility Support Representatives
(FS Reps) and/or Radiological Control Technicians (RCTs).

Managers effectively communicate their expectations such that employees
understand their roles and responsibilities for self-assessment. ESH&Q
PMP and Division SA Plans state, for example, that managers ensure that
R2A2s and individual performance goals of direct reports reflect
expectations regarding self-assessment activities and are aligned with
achieving organizational and institutional objectives and measures. The
QMO draft SA Plan states that the "QMO Manager will assign personnel to
perform" assessment activities. **(-)** ESH&Q PMP document does not
assign responsibility for preparation of document. Managers and staff
interviewed clearly understand their responsibilities for
self-assessment.

Scheduled assessment activities evidence the appropriate incorporation
of independent assessments and peer reviews. The RCD SA Plan (Section
4.5) states, "Where applicable, the Peer Review process, external
expertise and/or 'Subject Matter Experts' will be utilized for
assessment activities." **(+)** RCD has RCTs from one facility/area
conduct surveillances in another facility thus providing independent
peer review.

Assessment activities range from inspections, audits, and surveillances,
to periodically monitoring financial information. ESH&Q organizations
also use customer groups and customer surveys to measure performance.
Organizations use management and staff meetings as a means of measuring
progress. There are numerous examples of performance indicators and/or
metrics used throughout the Directorate. Activities also include
management walkthroughs and observations of work in progress. Some
organizations include benchmarking activities (e.g., ASTD) in their SA
Programs.

Tier I inspections and other surveillance-type activities are typically
designed to include workspace inhabitants to improve the communication
and understanding of strengths and areas for improvement. Many personnel
in the ESH&Q Directorate conduct assessment activities as a regular part
of their work; therefore, work activities of ESH&Q personnel are not so
much impacted rather their activities may impact other personnel.

3.  **Management and staff involvement commensurate with their
    responsibilities is evident.**

Planning involves managers and staff members as appropriate to ensure
that all aspects of the organization's operations are evaluated. SHSD
and EWMS SA Plans (Section 2.2.1) state that Self-Assessment
Coordinators are responsible for "developing the self-assessment plan by
interacting with the applicable subject matter experts (SME) and program
managers." Reinforcing this (Section 2.3), individuals such as SMEs,
Section/Program Managers, QA Representatives, etc., are assigned the
responsibility to provide input to self-assessment planning. The T&Q SA
Plan also documents these responsibilities for Program Managers, SMEs,
and staff. The RCD SA Plan (Section 3.2.2) documents that RCD Section
Managers are responsible to "plan and implement management system
assessments." Managers and staff members interviewed acknowledged their
involvement in development and/or review of both personal and
organizational goals and objectives.

**(-)** Some organizational managers did not review and approve
Self-Assessment Plans according to BNL's required schedule. Managerial
and organizational changes discussed previously contributed to these
delayed approvals. Initial versions of RCD and T&Q SA Plans were
approved prior to 12/31/03.

4.  **Key supporting organizational processes (tracking/trending
    systems, causal analysis, critiques, etc.) and tools are
    developed.**

The organization has effective processes for analyzing and trending the
results of assessment activities. The ESH&Q PMP (Section 4.0) states,
"Accomplishments against the objectives and measures of this plan are
reviewed approximately quarterly at regularly scheduled bi-weekly
managers meetings." This requirement is flowed down as in the SHSD SA
Plan (Section 4.1), which states, "Individuals responsible for
coordinating or conducting the assessment activities shall present the
results and make recommendations to the SHSD Management Team, usually
through distribution of the assessment report or at SHSD Management Team
Meetings. The management team shall review the information and
recommendations and determine appropriate corrective and improvement
actions." The PMP lists a Directorate Strategic Imperative to "fully
deploy tracking/trending process." Evidence of initial stages of this
deployment exists in the form of the Occurrence Reporting and Processing
System (ORPS) Quarterly Performance Analysis and Summary issued by QMO.

Organizations have processes in place for tracking corrective,
preventive, and improvement actions to completion. SA Plans document,
for example, that "Corrective and improvement actions identified through
assessment activities shall be managed in accordance with ESH Standard
1.2.1, Corrective Action Management and Tracking for \[External and
Internal\] Assessments. Corrective Actions associated with lab-wide
programmatic assessments shall be entered and tracked in the
Institutional level Assessment Tracking System (ATS). Internal
\[organizational\] corrective actions should be tracked to closure
through the Family Assessment Tracking System (FATS)." Actions resulting
from external assessments are generally tracked in Institutional ATS;
some assessments listed on the FY04 BHSO ES&H Assessment Schedule are
tracked on FATS instead of Institutional ATS. All ESH&Q organizations
use FATS for tracking actions resulting from internal assessment
activities. **(-)** Effectiveness of the process for tracking actions to
completion is in question considering that in Institutional ATS (as of
8/26/04), there were 40 actions overdue by ten days or longer (some
since November 2003), of which 28 actions (70%) are either owned by
ESH&Q or are part of assessments owned by ESH&Q. This review did not
include a determination of overdue actions in FATS. Additionally some
interviewees complained that corrective actions are assigned to them via
ATS without their knowledge, which is not in conformance with direction
provided in ESH 1.2.1.

Records are maintained that demonstrate effective planning, timely
completion of assessments, and analysis of results. SA Plans are
maintained as internally controlled documents. Examples of assessment
reports, action plans, SA Plan status reports, and year-end reviews
provided ample evidence that activities are appropriately planned and
completed, and that strengths, areas for improvement, and corrective
actions are determined.

Available information systems are used to effectively support assessment
activities. SA Plans and related procedures are typically maintained on
websites and/or shared computer drives. ESH&Q organizations all use FATS
to track assessments and/or actions. RCD, for example, uses FATS to
schedule quarterly self-assessments conducted by FS Reps and RCTs. SHSD
has instituted Compliance Suite (shared database) to enable BNL
organizations to track/trend injury and illness statistics and to track
actions items resulting from the 2003 OSHA inspection. The T&Q Program
Office uses a web-based customer survey.

5.  **Organizational plans/procedures address regulatory and/or SBMS
    drivers for assessment activities.**

Assessment activities are based on and linked to the organization's
performance objectives and measures as well as the BNL Critical Outcomes
and associated Performance Objectives/Measures. Attachments to the ESH&Q
PMP, the SHSD SA Plan, the EWMS SA Plan, and the T&Q SA Plan
specifically link organizational objectives with assessment activities,
indicators, measures, and/or metrics. The draft QMO SA Plan provides
this linkage in Section 2.3. The RCD SA Plan (Section 8.0) provides
explicit linkage only to 10 CFR 835, NRC and OSHA compliance
inspections, and BNL Critical Outcomes. **(-)** The RCD Annual
Self-Evaluation lists organization-specific Performance Measures,
Current Initiatives, Program Weaknesses, and Areas of Emphasis in FY
\[04\], but the RCD SA Plan does not provide linkage to these items.

All applicable "Required Assessments" are identified in some SA Plans
demonstrating appropriate consideration of integration of SBMS
Management System requirements. EWMS and SHSD SA Plans explicitly
include BNL-Required Assessments under Criterion 6.0, Compliance with
Laws, Regulations and Contractual Requirements. **(-)** There is no
explicit mention of these BNL-Required Assessments in QMO, RCD, or T&Q
SA Plans. However, since the ALD for ESH&Q is Steward for ten SBMS
Management Systems and ESH&Q Level 2 managers are points-of-contact for
these Management Systems, there is evidence of "appropriate
consideration of integration of SBMS Management System requirements"
into organizational and Management System SA Programs.

### Criterion 2: Deployment of Self-Assessment

Assessment activities are conducted as scheduled. Schedules may be
adjusted based on changing priorities as determined by organizational
managers. Results of assessment activities are documented and
communicated to responsible ESH&Q managers for review and analysis. Key
assessment results are regularly discussed and analyzed at
management/staff meetings. Managers evaluate assessment results to

identify both strengths and opportunities for improvement. Necessary
improvement actions are identified and communicated to responsible
managers and stakeholders.

1.  **Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.**

Documented year-end self-evaluations and/or quarterly SA Plan updates
provide evidence that assessment activities have been completed on
schedule. **(-)** The T&Q SA plan does not contain a schedule for
activities other than those completed on an annual basis (i.e., by end
of fiscal year). **(-)** The T&Q SA Plan also does not assign all
activities to specific personnel. Some adjustment in schedules was
necessary in SHSD due to the need for SHSD to support the OSHA
inspection of BNL including costing and prioritizing corrective actions.
Assessment activities have been rescheduled as indicated in quarterly
update.

SA Plans are updated to reflect changes in scope and/or schedule. RCD
and T&Q SA Plans have been updated since original issuance in FY04. The
ESH&Q PMP and draft QMO SA Plan are either recently issued or not yet
finalized/approved. Quarterly EWMS and SHSD SA Plan updates also
document changes to schedules and/or scope of activities.

Customer feedback is solicited as part of some organizations' assessment
activities. The EWMS SA Plan documents customer feedback strategies and
indicators under Assessment Criterion 3.0, Customer Focus and
Satisfaction. **(-)** The SHSD SA Plan also incorporates an objective
under Criterion 3.0 but does not specifically solicit customer feedback
as part of the strategies to support the objective. T&Q uses a customer
survey and solicits feedback for both web and classroom courses. T&Q
also interacts with BNL customers through the T&Q Steering Committee.
The RCD SA Plan (Section 3.1.3) lists a Division Manager responsibility
to "team with the BHSO counterpart in determining the scope of any
collaborative assessments scheduled for the upcoming fiscal year."
**(-)** The RCD SA Plan does not contain any mention of solicitation of
customer feedback although the RCD FY 2003 Self-Evaluation (4. Current
Initiatives) states, "RCD will focus on customer service; e.g., routine
meetings between business group managers and customer points of
contact." Interviewees from RCD acknowledged attending these meetings
and receiving other forms of feedback. RCD FY 2003 Self-Evaluation
(Section 7. Areas of Emphasis in FY \[04\]) includes Customer Service
with mention of using "customer satisfaction surveys" and "routine
face-to-face meetings with primary department points of contact," but
these activities are not listed in the RCD SA Plan. **(-)** The draft
QMO SA Plan includes quarterly QA Representative Meetings but does not
indicate that these meetings include solicitation of customer feedback.
All managers interviewed acknowledged routine contact with customers.

2.  **Assessments are documented and communicated as planned.**

ESH&Q organizations routinely document assessment results and distribute
these to responsible managers and staff. Independent and/or external
assessment results are also routinely provided to appropriate personnel.

ESH&Q managers develop, review, and/or approve assessment schedules.
Assessments are timed, for example, to serve as input to BNL's Annual
Self-Evaluation Report, ISO 14001 Recertification, EMS Management
Reviews, Quarterly Triennial Assessments, and year-end organizational
self-evaluations.

3.  **Assessment results (internal and external) are evaluated/analyzed
    to a degree commensurate with the type of assessment. Strengths are
    identified. Corrective/improvement actions are identified,
    prioritized, assigned to specific owners, and assigned due dates.
    All conditions are tracked to closure. Actions are validated as
    improving performance.**

ESH&Q has an effective process for identifying opportunities for
improvement. Assessment reports reviewed identify strengths and areas
for improvement (concerns, findings, observations, recommendations,
nonconformances, and noncompliances). Year-end self evaluations
specifically identified opportunities for improvement.

While it is clear that assessment activities are linked with prioritized
objectives (Critical Outcomes, Lab Strategic Imperatives, Directorate
Strategic Imperatives, and Organizational Strategic Imperatives), it is
not as clear that improvements (i.e., corrective actions) are routinely
prioritized accordingly. In other words, it is not clear that actions
resulting from higher priority assessments are given higher priority.
Year-end self evaluations do provide evidence that some improvements
identified in the previous fiscal year are appropriately carried over to
the next fiscal year to become the topic of additional objectives,
measures, and assessments. Actions developed in response to the OSHA
inspection were "risk-ranked."

According to the various SA Plans, individuals who coordinate or conduct
assessment activities have the responsibility to "evaluate" assessment
results and recommend corrective actions. Some organizations then use a
"management team" to "determine appropriate corrective and improvement
actions." The IO Reviewer found no explicit reference to causal analysis
or consideration of causal factors in SA Plans. However, if one assumes
that higher levels of hazard/risk are represented by items reportable to
ORPS or NTS, for example, then the Corrective and Preventive Action
subject area requires that more formal causal analysis be conducted. The
IO Reviewer found explicit reference to the Corrective and Preventive
Action subject area in only the RCD SA Plan.

**(-)** Change control for some action due dates has not been timely as
discussed in Section 2.1.4 previously. There is evidence that due dates
for other items have been managed with recognition of priorities as with
the assessments postponed due to supporting the OSHA inspection.

Some improvements are validated as part of periodic reviews, such as
Tier I inspections and Triennial Assessments. Some performance measures
are designed to detect any adverse trends in areas that have previously
been corrected/improved. Such measures include lost work day cases,
traffic violations, first aid cases, turnaround time for design review
requests, required personnel training percentages, customer
satisfaction, and field observations. ESH&Q managers have also requested
that follow-up assessments of effectiveness of actions be conducted by
the IA&O Office, for example.

4.  **Evidence of timely self-identification of issues exists.
    Significant issues are brought to the attention of management and
    disclosed to regulatory/oversight agencies in accordance with
    contractual obligations.**

Results are analyzed to identify potential noncompliances with nuclear
safety rules (Price-Anderson Amendments Act). The BNL PAAA Coordinator
routinely receives Radiological Awareness Reports, Nonconformance
Reports, and reports of radiological assessments to review for
noncompliance with nuclear safety rules. SA Plans require that the
management team "determine the need to report issues identified through
ESH&Q assessment activities to the Laboratory Price Anderson Amendments
Act Coordinator." **(-)** It is noted that EWMS, SHSD, and RCD SA Plans
erroneously state that activities shall be reported to the PAAA Working
Group instead of the PAAA Coordinator.

Results are analyzed to determine potential impacts on SBMS documents.
ESH&Q organizations identify some assessment activities as "horizontal"
or programmatic assessments. These assessments are designed to measure
the effectiveness of implementation of SBMS Management Systems across
the Laboratory and results from these assessments have direct impact on
SBMS documents. ESH&Q managers who are Management System Stewards and/or
Points of Contact have developed Management System Templates that
describe assessment activities developed to measure the effectiveness of
Management Systems. EWMS and SHSD SA Plans, for example, incorporate a
number of assessment activities specifically addressing SBMS subject
areas and ES&H Standards, for example. The RCD SA Plan attaches the
Template for Radiological Control Management System.

Results are appropriately considered for publication in the BNL Lessons
Learned Program. SA Plans require that the management team determine
"the need to submit assessment results to the Lab-wide Lessons Learned
program." Events or conditions that are reported to ORPS come to the
attention of the BNL Lessons Learned Coordinator and may result in the
issuance of a Lab-wide Lesson Learned. ESH&Q personnel have authored
many BNL Lessons Learned receiving acknowledgement from BNL management
for their efforts. **(+)** ESH&Q communicates lessons through the ES&H
Monthly Summary, EWMS Division Highlights, and the BNL Radiological
News. ESH&Q organizations also communicate lessons learned information
through meetings with ES&H Coordinators, Quality Assurance
Representatives, EMS Representatives, and the Generator Information
Group.

5.  **Management involvement is evident.**

ESH&Q managers routinely participate in planning and conducting
assessment activities. Managers participate in the planning of
assessment activities as required in SA Plans (see discussion in Section
2.1.3). The following paragraph provides specific examples of management
involvement in conducting assessments. Managers conduct periodic reviews
of budget/spending. Managers interact routinely with customers receiving
solicited and unsolicited feedback. Managers also regularly participate
in management team and staff meetings.

**(+)** There is an evident increase in the observation of work
activities by both managers and subject matter experts throughout the
Directorate since the previous IO Review. The ALD has participated in
several Senior Management Work Observations led by the IA&O Office as
documented in reports prepared by IA&O. Managers of RCD and EWMS, for
example, provided documented results from observations conducted,
respectively, at the BGRR Restoration Project and at an off-site
environmental sampling activity at a neighboring lake. T&Q subject
matter experts are conducting observations of equipment operation to
verify that trained personnel were appropriately implementing their
training. Managers also conduct unscheduled walkthroughs, which may not
be documented, and participate to varying degrees in Tier I inspections.

ESH&Q managers participate in the analysis and evaluation of assessment
results. Management team meetings at both the Directorate and
Division/Office levels are used to evaluate assessment results. Managers
participate in analysis of higher risk (more visible) events and
conditions. Managers are routinely involved in development and/or review
of action plans responding to assessment activities.

Managers verify that improvements are implemented. ESH&Q managers are
listed as Assessment Owners in ATS and FATS thus accepting
responsibility for verifying closure of actions. **(-)** Given the
previous discussion (Section 2.1.4) regarding potentially ineffective
corrective action management, there is some concern that absent a
specific requirement in SA Plans or individual assessment activities to
verify/validate previous actions, this is not routinely accomplished.
Verification of some improvements is included as part of periodic
assessment activities, such as Tier I inspections, EMS recertification,
and Triennial Assessments. ESH&Q managers' year-end evaluations of
individual and organizational performance also serve to verify that
improvements have been effectively implemented.

### Criterion 3: Results from Self-Assessment

The ESH&Q Directorate provided timely input to the FY03 BNL Year End
Self-Evaluation Report as in previous years summarizing performance
results for selected Objectives/Measures under Critical Outcome 3.0. RCD
and T&Q prepared year-end self-evaluation reports. The Directorate also
critically documented performance in the ESH&Q and BNL-wide EMS
Management Review Meeting Minutes. The ESH&Q Budget Presentation in May
2004 served to update BNL Senior Management on FY04 performance and FY05
plans. There is appropriate linkage between the ESH&Q SA Plans and BNL
Contract Measures and Strategic Imperatives.

1.  **Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations, such as mission
    achievement and retention/expansion of core competencies.**

Progress towards achievement of near-term and long-term goals and
objectives is evident. Quarterly updates, year-end reports, and
presentations (EMS Management Reviews and Annual Budget Review) document
progress. Examples include the Triennial Recertification of BNL under
ISO 14001, the recent achievement of membership in EPA Performance
Track, completion of initial phases of OHSAS 18001 registration, DOELAP
accreditation of Personnel Monitoring, completion of BNL Safety
Improvement Plan, completion of Nuclear Strategic Plan,
reorganization/merger of some ESH&Q Divisions/Offices, completion of the
prioritized list of OSHA findings with cost estimate, completion of
initial phases of Lab-wide tracking/trending program supporting
performance analysis, implementation of Compliance Suite, completed
transition of Long Term Remedial Action (LTRA) function to EWMS, review
and update of Subject Areas and procedures, and review and update of
training materials.

It is evident that ESH&Q services and products have been enhanced.
**(+)** Procurement and maintenance of certifications (ISO 14001,
DOELAP, OHSAS 18001) indicate improvement in Laboratory operations and
support functions. Organizations have benchmarking and customer service
activities underway but have not yet determined results from these.

Evidence exists that customers and other stakeholders value the products
and services of the organization. T&Q formally measures customer
satisfaction and reported a 93% approval rating in FY 2003. Customers
from research organizations expressed strong satisfaction for the
services of Environmental Compliance Representatives. Customers also
expressed satisfaction with the support provided by FS Representatives
and Radiological Control Technicians. Customers from the two research
organizations reviewed in FY 2004 favorably acknowledged the support
provided by Industrial Hygiene (IH) SMEs but expressed an interest in
having greater IH participation on Tier I inspections. Research
organizations also expressed satisfaction with support provided by QMO
personnel (ATS, ORPS, QA Plan development).

It was difficult to identify specific examples of ESH&Q improvements
specifically resulting from the self-assessment process given the amount
of "external" assessments conducted of ESH&Q functional areas. However,
ESH&Q is to be credited with implementing improvements regardless of how
the need for improvement was identified to produce the examples
discussed in previous paragraphs. It is also noted that the
self-assessment process has improved as managers became more involved in
the process, managers and staff became more clearly aware of their
responsibilities now specifically delineated in SA Plans, and all ESH&Q
organizations used FATS for scheduling activities and tracking results.

2.  **Evidence exists that there is an appropriate connection between
    results of organizational self-evaluation and development of
    strategic/institutional plans.**

ESH&Q has not yet developed a strategic plan for the Directorate nor
have individual organizations developed strategic plans or business
plans. This area for improvement was identified to ESH&Q as part of IO
Report SA 02-02 based on the documented intent to develop an ESH&Q
Strategic Plan in FY02. ESH&Q PMP and SA Plans evidence consideration of
Laboratory Strategic Imperatives/Initiatives. ESH&Q provides input to
the BNL Critical Outcomes and Institutional Plan to ensure appropriate
connection with BNL needs.

Improvement actions are either acted upon at the
organization/Directorate level or communicated to BNL senior management
to be included in the Laboratory's improvement agenda, as appropriate.
Critical Outcome related performance and areas for improvement are
communicated to BNL management in the ESH&Q contributions to BNL's
Annual Self-evaluation.

ESH&Q organizations demonstrated responsiveness to some of the Areas for
Improvement and the Recommendations included in **IO Report SA 02-02**:

-   **Some SA Plans were not completed by the end of the first quarter
    of FY01 (December 2001) as expected by Laboratory management.** This
    has been partially addressed since RCD and T&Q managers approved SA
    Plans prior to December 31, 2003. Organizational and management
    changes in the ESH&Q Directorate including EWMS, SHSD, and QMO may
    have resulted in late issuance of FY04 SA Plans.

-   **The RCD SA Plan does not include an explicit listing of
    organizational performance objectives.** This has not been resolved.
    As noted previously, RCD's FY04 SA Plan does not include objectives
    other than compliance, while RCD's FY03 Annual Self-Evaluation
    documents other performance measures including specific expectations
    for Facility Support, Analytical Services Laboratory,
    Instrumentation and Calibration, Personnel Monitoring, Radiological
    Assistance Program, and Health Physics Technical Support.

-   **ESH&Q organizations have not involved SMEs from the T&Q Office in
    assessments.** IO recommended that ESH&Q managers ensure that SMEs
    from T&Q are involved as appropriate in assessment activities
    wherein effectiveness of training may be an issue. Independent
    assessments of electrical safety conducted by IO did involve an
    individual from T&Q. T&Q FY03 self-evaluation report states, "Though
    the T&Q Program Office solicited program owners for areas in which
    we should perform field evaluations to ensure training
    effectiveness, there were no areas identified this year for this
    activity." T&Q further indicated aggressive solicitation of program
    owners in FY04. Performance Measure TQ 9.0, Field Assessment of T&Q
    Training Courses, tracks the conduct of observations of forklift
    operators during FY04.

-   **ESD and SHSD SA Coordinators did not have ready access to their
    respective FATS Report pages.** This has been addressed. EWMS and
    SHSD SA Coordinators demonstrated ready access to their respective
    FATS. Individuals from other ESH&Q organizations also demonstrated
    such access.

-   **ESH&Q has not developed a Strategic Plan.** During interviews the
    ALD and other organizational managers articulated intent to develop
    ESH&Q and organization-specific strategic/business plans, but these
    have not yet been developed.

## Conclusions

The ESH&Q Directorate evidences improvement in the organizational
Self-Assessment Program since FY02 over what was already recognized as
an effective program. The SA Program is largely comprehensive in scope
with all organizations preparing SA Plans. Some organizations do not
incorporate the solicitation of customer feedback into their SA Plans.
Approaches to self-assessment activities include more than just
traditional audits and represent a performance-based concept. Roles and
responsibilities for self assessment are well defined and communicated.
Processes for documenting results of activities, analyzing results,
identifying opportunities for improvement, and tracking actions are in
place. There is evidence that corrective action management needs
improvement in that ESH 1.2.1 is not fully implemented. The ESH&Q PMP
and organizational SA Plans document an annual cycle of continuous
improvement in effectiveness and efficiency, represent a forward-looking
approach, and provide linkage with the BNL Strategic Imperatives and
Institutional Plan in planning and achievement of longer-term goals and
objectives.

### Strengths 

-   The merger process undertaken by the then recently-formed EWMS
    Division represents an exemplary process for developing a
    Self-Assessment Program/Plan. EWMS personnel developed the FY04 SA
    Plan by building on the best elements of the SA Plans of the former
    ESD and WMD. The step-by-step process as documented in meeting
    minutes provides an effective model for developing a comprehensive
    SA program.

```{=html}
<!-- -->
```
-   EWMS and SHSD SA Plans provide an exemplary format for
    organizational SA Plans. The SA Plans describe the overall SA
    process including clearly defined roles and responsibilities. The
    attachments provide clear linkage to Institutional, Directorate, and
    Divisional imperatives. The attachments are organized according to
    the IAP (Baldrige) framework thus ensuring a comprehensive
    perspective of organizational performance.

-   SHSD's Plan explicitly includes priorities for each assessment
    activity and states, "This ranking scheme allows the most effective
    use of limited Division resources."

-   There is an evident increase in the observation of work activities
    by both managers and subject matter experts throughout the
    Directorate since the previous IO Review.

-   RCD's self-assessment and surveillance activities evidence
    improvement since the last IO Review. The activities have become
    more connected to radiological control procedures and now involve
    RCTs as well as the FS Reps. Also FS personnel from one
    building/facility now conduct surveillances in other facilities thus
    providing **independent peer review of radiological performance**.

-   Procurement and maintenance of certifications (e.g., ISO 14001,
    DOELAP, OHSAS 18001) are indicative of improvement in Laboratory
    operations and support functions. DOE views recognition of effective
    performance by "third parties" as a key element of contractor self
    assessment. ESH&Q organizations have integrated certifications into
    their SA Plans.

-   The RCD Year-end Self Evaluation provides a comprehensive,
    self-critical analysis of performance for 2003. This document also
    provides an exemplary model of an organizational self evaluation in
    that it identifies strengths and areas for improvement and provides
    recommended actions.

```{=html}
<!-- -->
```
-   The ESH&Q Directorate uses several effective communication tools to
    provide ES&H information both within the Directorate and Lab-wide.
    Examples include ESD Highlights, WM Monthly, FS Monthly, Rad
    Newsletter, and the ESH Newsletter.

### Areas for Improvement 

-   Some ESH&Q organizational managers did not review and approve
    Self-Assessment Plans according to BNL's required schedule.
    Managerial and organizational changes discussed previously
    contributed to these delayed approvals.

> **Recommendation:** Managers should review and approve FY05 SA Plans
> by December 31, 2004, or as otherwise directed by any revisions to the
> Integrated Assessment subject area.

-   The draft QMO SA Plan does not explicitly include the Laboratory
    Strategic Imperative "Self-Assessment and Performance Measure
    Processes" as indicated in ESH&Q PMP Section 2.3.4. The draft QMO SA
    Plan does not include the "projectizing" of SBMS listed as a
    Directorate Strategic Imperative in Section 2.4 of the PMP.

> **Recommendation:** The QMO manager should ensure that the FY04 SA
> Plan is approved and that the approved version incorporates the
> referenced imperatives.

-   RCD's SA Plan does not include performance objectives other than
    compliance, while RCD's FY03 Annual Self-Evaluation documents other
    performance measures including specific expectations for sections
    within the Division.

> **Recommendation:** RCD should adopt the attachment format used in the
> EWMS and SHSD SA Plan attachments to ensure a comprehensive scope for
> self-assessment. The RCD manager should ensure that areas for
> improvement, Current Initiatives, Program Weaknesses, and Areas of
> Emphasis identified in year-end self-evaluations are addressed in
> subsequent SA Plans.

-   The T&Q SA plan does not contain a schedule for activities other
    than those completed on an annual basis and does not assign all
    activities to specific personnel.

> **Recommendation:** The T&Q manager should ensure that assessment
> activities are assigned to appropriate personnel and that these
> activities are clearly scheduled.

-   The ESH&Q PMP document does not assign responsibility for
    preparation of document.

> **Recommendation:** The ALD for ESH&Q should ensure that
> responsibility for preparation of the document is assigned.

-   As of 8/26/04, there were 28 actions overdue in Institutional ATS
    that are either owned by ESH&Q or are part of assessments owned by
    ESH&Q. There is also concern that absent a specific requirement in
    SA Plans or individual assessment activities to verify/validate
    previously completed actions, this is not routinely accomplished.

> **Recommendation:** ESH&Q managers should review their
> responsibilities as Assessment Owners in ESH Standard 1.2.1.
> Assessment Owners should review any corrective actions about to become
> overdue and determine whether or not a change of due date is
> appropriate. Assessment Owners should routinely verify that actions
> are completed prior to closing an assessment in ATS and should
> determine whether or not validation of effectiveness of completed
> actions should be conducted at some future time.

-   There is no explicit mention of the BNL-Required Assessments in QMO,
    RCD, or T&Q SA Plans.

> **Recommendation:** Managers of named organizations should ensure that
> required assessments applicable to their organizations (per the
> Integrated Assessment subject area exhibit "FY04 Required Line
> Self-assessments") are added to scheduled assessment activities in
> their SA Plans.

-   The SHSD SA Plan incorporates an objective under IAP Criterion 3.0
    but does not specifically solicit customer feedback as part of the
    strategies to support the objective. The RCD SA Plan does not
    contain any mention of solicitation of customer feedback. The draft
    QMO SA Plan includes quarterly QA Representative Meetings but does
    not indicate that these meetings include solicitation of customer
    feedback.

> **Recommendation:** Managers of named organizations should ensure that
> solicitation of customer feedback is included among scheduled
> assessment activities in their SA Plans. The EWMS SA Plan attachment,
> Criterion 3.0, provides a useful example of how to incorporate.

-   The EWMS, SHSD, and RCD SA Plans erroneously state that results of
    certain activities shall be reported to the PAAA Working Group
    instead of the PAAA Coordinator.

> **Recommendation:** Managers of named organizations should ensure that
> "Working Group" is revised to "Coordinator" in their SA Plans.

### IAP Management System Programmatic Issues

It is acknowledged that some of the issues discussed below are known to
BNL management and the IAP Management System Steward and Point of
Contact, and that efforts may be underway to address these issues. These
issues are noted here for emphasis.

DOE personnel have communicated to BNL management an expectation that
BNL's self-assessment process be made more credible, rigorous and
robust. It is recognized that DOE has not been specific as to how DOE
expects to measure these desired characteristics. The ALD for ESH&Q, as
Management System Steward for the Integrated Assessment Program, has
also indicated intent to improve BNL's self assessment process in the
FY04 PPM document; however, specifics of how to accomplish this intent
have not been documented in either the FY04 PPM or the QMO SA Plan. It
is recommended that the ALD for ESH&Q develop a plan for implementing a
more credible, rigorous and robust self-assessment process. The ALD
should review and revise the IAP Management System Description and the
Integrated Assessment subject area accordingly. Once characteristics of
the improved self-assessment process have been identified and
communicated across BNL, criteria for the review of organizational and
management system self-assessment programs should also be modified.

[]{#__RefHeading___Toc68064789 .anchor}Exhibit 1 - Documents Reviewed

1.  Environmental, Safety, Health and Quality Directorate Performance
    Management Program, 2004, Revision 0 (6/04/04)

2.  Brookhaven National Laboratory Safety Improvement Plan, Draft 5
    (7/12/04)

3.  Safety Improvement Plan and OSHA, \[Presentation to\] Management
    Council, Jim Tarpinian (3/16/04)

4.  Environmental, Safety, Health and Quality (ESH&Q) Directorate FY05
    Budget Presentation, James E. Tarpinian (5/07/04)

5.  Environmental & Waste Management Services Division Self-Assessment
    Plan FY 2004, Revision 0 (3/16/04)

6.  Environmental and Waste Management Services Division FY2003
    Accomplishments

7.  BNL Memorandum from M. Davis to G. Goode, EWMS Division
    Self-Assessment Program Quarterly Report -- 1^st^ and 2^nd^ Quarters
    FY2004 (5/12/04)

8.  Environmental Objectives & Suggested Targets FY 2004 (Approved by G.
    Goode -- 1/23/04)

9.  BNL Environmental Services Division Environmental Sampling Program
    Field Audit Checklist (7/19/04)

10. Meeting Minutes -- EWMS Self-Assessment Plan Merger (9/15/03)

11. Environmental & Waste Management Services Division Highlights, June
    2004

12. \[Draft\] EWMS Goals (1/30/04)

13. BNL Programmatic Self-Assessment Report of Liquid Effluents with
    BHSO Observation (6/30/04)

14. Assessment Tracking System #2152, EWMSD FY 2004 Self-Assessment Plan
    (Printed: 7/08/04)

15. Assessment Tracking System #2070, EWMS Website Updated (Printed:
    1/21/04)

16. ESH&Q/DO Environmental Management System FY03 Management Review
    \[Presentation\] (12/09/03)

17. BNL Environmental Management System Institutional-Level EMS
    Management Review \[Presentation\] (12/15/03)

18. Safety and Health Services Division Self-Assessment Plan FY 2004,
    Revision 0F (5/21/04)

19. Safety and Health Services Division Self-Assessment Plan -- 6/04
    Status

20. Safety and Health Services Family Assessment Tracking System,
    Assessment Summary Report (Printed: 7/13/04)

21. Safety and Health Services Division Staff Meeting Minutes, Report
    period Ending: 10/07/03

22. BNL Electrical Safety Program Self-Assessment FY03: Interlock Safety
    (7/31/03)

23. Corrective Actions Resulting from BNL Electrical Safety Program
    Self-Assessment FY03: Interlock Safety

24. FY 2003 Self-Assessment of the BNL Respiratory Protection Program,
    Corrective Action Plan (10/20/03)

25. Radiological Control Division Self-Assessment Plan FY 04, Revision 1
    (6/02/04)

26. BNL Memorandum from Charles W. Schaefer to Radiological Control
    Division, Fiscal Year 2003 Annual Self-Evaluation of the
    Radiological Control Division Performance (12/10/03)

27. Radiological Control Division Triennial Assessment Program, Revision
    1 (6/02/04)

28. E-Mail from C. Schaefer to F. Petschauer, et al., Surveillance of
    BGRR BGD Work Activities (6/22/04)

29. BNL Memorandum from Dennis Ryan to Charles Schaefer, Facility
    Support monthly report for May (6/25/04)

30. FS Reps Meeting Minutes for 6/15/04

31. BNL Memorandum from Dennis J. Ryan, Manager, Facility Support, to
    Qualified BNL Radiological Workers, Responsibilities associated with
    work conducted under a Radiation Work Permit (RWP) (6/25/04)

32. Radiological Control Assessment Query Results (Printed 7/09/04)

33. Training & Qualifications Program Office FY04 Self-Assessment Plan,
    December 2003 (Amended Signed 6/29/04)

34. BNL Memorandum from Ernest L. Tucker to Beth Schwaner,
    Self-Assessment Results for FY 2003 (10/29/03)

35. BNL Memorandum from Ernest L. Tucker to B. Schwaner, Customer Survey
    Results for FY 2003 (8/13/03)

36. Training & Qualifications Program Survey -- 2004

37. Operational Efficiency Review Work Group Template, Work Group:
    Training, Function/Service Under Review: BNL Training Program
    (Draft)

38. BNL Memorandum from K. Krasner to T. Monahan, FY 2003
    Self-Assessment of the BNL Contractor/Vendor Orientation (CVO)
    Training Program (9/24/03)

39. E-Mail from B. Schwaner to R. Selvey, et al., Lesson Learned for
    Training (1/07/04)

40. E-Mail From B. Schwaner to Level I Managers, et al., Training Survey
    and July Report (6/30/04)

41. Job Training Assessment \[Survey Form\]

42. BNL Memorandum from Ernest L. Tucker to Beth Schwaner, Data Entry
    Summary for Third Quarter of FY 2004 (7/12/04)

43. Training & Qualifications Family Assessment List (Printed: 7/15/04)

44. \[Draft\] Quality Management Office Self Assessment Plan fiscal Year
    2004, Revision 1 (7/01/04)

45. Quality Programs and Services Office Self Assessment Plan for FY2003
    (with status -- 7/14/04)

46. Quality Management -- A Strategic Vision for a Diverse Environment
    \[presentation\]

47. Quality Management Office (QMO) FY05 Budget Presentation (5/07/04)

48. QMO Staff meeting \[minutes\] (5/25/04)

49. BNL Memorandum from R. Savage to J. Tarpinian, Internal Controlled
    Documents Programmatic Assessment -- Final Report (3/19/04)

50. BNL Memorandum from J. E. Tarpinian to All Employees, Safety Alert
    (6/14/04)

51. Occurrence Reporting & Processing System (ORPS) Quarterly
    Performance Analysis and Summary, 2nd Quarter CY 2004 (Distributed:
    8/17/04)

[]{#__RefHeading___Toc68064790 .anchor}Exhibit 2 -- Personnel
Interviewed

1.  J. Tarpinian, ALD for Environment, Safety, Health and Quality

2.  G. Goode, Manager, Environmental and Waste Management Services
    (EWMS) Division

3.  S. Hoey, Interim Manager, Safety and Health Services Division (SHSD)

4.  R. Lebel, Manager, Quality Management Office (QMO)

5.  C. Schaefer, Manager, Radiological Control Division (RCD)

6.  B. Schwaner, Manager Training and Qualifications (T&Q) Program
    Office

7.  B. Penn, Manager, ESH&Q Business Operations Management

8.  M. Davis, EWMS SA Coordinator

9.  R. Travis, SHSD SA Coordinator

10. P. Heotis, RCD Quality Representative

11. E. Tucker, T&Q SA Coordinator

12. S. Coleman, EWMS

13. K. Conkling, RCD

14. J. Ellerkamp, SHSD

15. W. Fortunato, T&Q

16. P. Harrington, T&Q

17. H. Kahnhauser, RCD

18. K. Krasner, SHSD

19. J. Odin-McCabe, RCD

20. C. Ogeka, ESH&Q Business Operations

21. P. Pohlot, EWMS

22. D. Rocco, EWMS

23. D. Ryan, RCD

24. J. Selva, EWMS

25. R. Selvey, SHSD

26. E. Sierra, QMO

27. C. Weilandics, SHSD

28. J. Wilke, QMO

[]{#__RefHeading___Toc68064791 .anchor}Exhibit 3 - Documents Referenced

1.  SBMS Management System Description: Integrated Assessment Program,
    Issue Date: July 1999

2.  BNL Independent Oversight Office FY 2004 Program Plan: Review of
    Organizational Self-Assessment Programs (December 30, 2003)

3.  U.S. Department of Energy Contract with Brookhaven Science
    Associates, DE-AC02-98CH10886

4.  Appendix B (to DE-AC02-98CH10886), Critical Outcomes, Objectives,
    and Performance Measures FY 2004 Brookhaven National Laboratory
    (Effective: 03/16/04)

5.  Year End Self-Evaluation - Brookhaven National Laboratory - Fiscal
    Year 2003 (December 19, 2003)

6.  U.S. Department of Energy Fiscal Year 2003 Annual Evaluation of
    Brookhaven National Laboratory (April 19, 2004)

7.  Brookhaven National Laboratory Draft Institutional Plan FY 2004 --
    FY 2008, September 2003

8.  SBMS Subject Area: Integrated Assessment, Effective Date: January
    2004

9.  SBMS Subject Area: Corrective and Preventive Action, Effective Date:
    December 2003

10. SBMS ESH Standard 1.2.1, Corrective Action Management and Tracking
    for External and Internal Assessments, Rev. 5, Effective Date:
    December 2002

11. DOE-BAO FY 04 ES&H Assessment Plan (October 1, 2003)

12. Brookhaven National Laboratory Independent Oversight Report SA
    02-02, Review of Self-Assessment Programs of the Environment,
    Safety, Health and Quality Directorate (September 11, 2002)