Brookhaven Science Associates

Brookhaven National Laboratory

Internal Audit and Oversight Report

SA 04-02

Review of the Self-Assessment Program

of the

Energy, Environment and National Security Directorate

July 30, 2004

Prepared by:

John Usher, Reviewer Date

> Independent Oversight

Approved by:

Robert McNair, Manager Date

Independent Oversight

Contents

1.0 Introduction [1](#introduction)

1.1 Purpose and Scope [1](#purpose-and-scope)

1.2 Review Process [1](#review-process)

1.3 Review Criteria [2](#review-criteria)

2.0 Program Summary [3](#program-summary)

2.1 Criterion 1: Approach to Self-Assessment
[3](#criterion-1-approach-to-self-assessment)

2.2 Criterion 2: Deployment of Self-Assessment
[7](#criterion-2-deployment-of-self-assessment)

2.3 Criterion 3: Results from Self-Assessment
[9](#criterion-3-results-from-self-assessment)

3.0 Conclusions [12](#conclusions)

3.1 Strengths [12](#strengths)

3.2 Areas for Improvement [13](#areas-for-improvement)

3.3 IAP Management System Programmatic Issues
[14](#iap-management-system-programmatic-issues)

**Exhibits**

Exhibit 1 - Documents Reviewed [15](#__RefHeading___Toc68064789)

Exhibit 2 -- Personnel Interviewed [17](#__RefHeading___Toc68064790)

Exhibit 3 - Documents Referenced [18](#__RefHeading___Toc68064791)

## Introduction

### Purpose and Scope

This report presents the results of Internal Audit and Oversight's
review of the Self-Assessment (SA) Program of the Energy, Environment
and National Security (EENS) Directorate. As an element of the
Integrated Assessment Program (IAP), Internal Audit and Oversight (IA&O)
is chartered to independently verify the effectiveness of Brookhaven
National Laboratory's Self-Assessment Program. To accomplish this, IA&O
personnel perform reviews of Directorates or other organizations
reporting to the Laboratory Director in accordance with the requirements
and guidelines provided in the IAP Management System Description.
Reviews are identified and scheduled in the FY 2004 Program Plan: Review
of Organizational Self-Assessment Programs \[FY04 IO Review Plan\].

FY04 reviews focus on the organization's "approach" to self-assessment,
the status of "deployment," and the "use of results" to improve
performance. Specifically, IA&O will review how the organization is
planning to implement the self-assessment program, how the organization
conducts the self-assessment program, and how the results from
self-assessment are analyzed and used to improve performance. In the
course of this review, comparison of the current status of the EENS SA
Program to that reported in **Independent Oversight Report SA 01-01,
Evaluation of Self-Assessment Programs of the**

**Energy, Environment and National Security Directorate and the Office
of Intellectual Property and Industrial Partnerships \[IO Report SA
01-01\]** was performed to measure progress since the FY 2001
evaluation.

### Review Process

This review formally began on March 31, 2004, with an inbriefing
conducted with personnel from EENS. This meeting served to introduce the
IA&O Reviewer, John Usher, who provided information on review
objectives, methodology, review criteria, and schedule. The inbriefing
also served to inform the IA&O Reviewer about EENS and to identify key
personnel involved in the planning and implementation of
self-assessment.

The following data and collection methods were used during the review:

-   Documents Reviewed (Exhibit 1)

```{=html}
<!-- -->
```
-   Interviews (Exhibit 2)

```{=html}
<!-- -->
```
-   Documents Referenced (Exhibit 3)

Interviews were conducted using prepared questions tailored to address
the review criteria (see Section 1.3) in terms of key business
factors/performance objectives. Notes were prepared during each
interview to record the information gathered. Comments, concerns, and
follow‑up actions were also documented after each interview.

An outbriefing was conducted on July 14, 2004, with personnel from EENS.
At this meeting, the IA&O Reviewer summarized results of the review and
submitted a draft report for review by EENS personnel. The meeting also
gave EENS personnel the opportunity to provide feedback to the IO
Reviewer on the review process.

### Review Criteria 

The IA&O assessment process evaluates an organization's self-assessment
program against the criteria published in the **FY04 IO Review Plan**.
These criteria are:

1.  Approach

```{=html}
<!-- -->
```
1.  The scopes of planned assessment activities are comprehensive, have
    a strong technical basis, and are balanced with work activities.

2.  The methods for conducting key scheduled assessments are defined and
    are commensurate with types of assessments planned and performance
    information desired.

3.  Management and staff involvement commensurate with their
    responsibilities is evident.

4.  Key supporting organizational processes (tracking/trending systems,
    causal analysis, critiques, etc.) and tools are developed.

5.  Organizational plans/procedures address regulatory and/or Standards
    Based Management System (SBMS) drivers for assessment activities.

```{=html}
<!-- -->
```
2.  Deployment

 

1.  Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.

2.  Assessments are documented and communicated as planned.

3.  Assessment results are evaluated/analyzed to a degree commensurate
    with the type of assessment. Strengths are identified.
    Corrective/improvement actions are identified, prioritized, assigned
    to specific owners, and assigned due dates. All conditions are
    tracked to closure. Actions are validated as improving performance.

4.  Evidence of timely self-identification of issues exists. Significant
    issues are brought to the attention of management and disclosed to
    regulatory/oversight agencies in accordance with contractual
    obligations.

5.  Management involvement is evident.

```{=html}
<!-- -->
```
3.  Results

```{=html}
<!-- -->
```
1.  Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations, such as mission
    achievement and retention/expansion of core competencies.

2.  Evidence exists that there is an appropriate connection between
    results of organizational self-evaluation and development of
    strategic/institutional plans.

Further detailed information supplementing the review criteria is
provided in the **FY04 IO Review Plan**
(<http://www.io.bnl.gov/safy04.doc>).

## Program Summary

This review covers the EENS Directorate comprising the Energy Sciences
and Technology (EST) Department, the Environmental Sciences (ES)
Department, the Nonproliferation and National Security (NNS) Department,
and the Center for Data Intensive Computing (CDIC).
Organizational/functional areas addressed in this review included the
Research Operations (RO) Office and the Business Operations (BO) Office.
The Associate Laboratory Director (ALD) for EENS does not have
stewardship responsibilities for any SBMS Management Systems.

A discussion and analysis of data collected for the review of the EENS
Directorate is presented for each criterion. Areas that demonstrated
positive performance or programmatic strengths (criteria exceeded) are
identified with a **(+)**. Areas where criteria are met but not exceeded
are unmarked. Areas that indicated opportunities for improvement
(criteria not met or partially met) are noted using a **(-)**.

### Criterion 1: Approach to Self-Assessment

The Deputy Manager, RO, prepared the CY 2004 Self-Assessment Plan \[SA
Plan\] for the Research Operations Office. The Manager, RO, reviewed and
the ALD approved the SA Plan. The SA Plan, Rev. No. 5, is dated April 1,
2004. The revision number reflects revision of the CY 2003 SA Plan; the
CY 2004 SA Plan was issued on April 1, 2004. One Department Chair
provided a draft SA Plan (April 14, 2004) that had not been finalized.
There are no other SA Plans, per se, existing in the EENS Directorate.

The scope of the SA program is largely comprehensive in addressing the
Directorate's research, operations, and support services. Research is
addressed in Departmental and Directorate Strategic Plans and in
Laboratory and Directorate Strategic Initiatives documents. Financial
management activities such as budget planning and ensuring adherence to
spending plans are evident as documented in the Fiscal Year End
Analyses. Appropriate methodologies for assessment are described in the
RO SA Plan. Assessment activities are appropriately prioritized.
Managers, staff, and key stakeholders are appropriately involved in
planning assessment activities. Processes for supporting the
organizational SA programs are in place, although Directorate personnel
do not use the Family Assessment Tracking System (ATS) available to
them. Evidence exists that organizational managers appropriately
consider SBMS, contractual, and regulatory drivers in developing
performance goals/objectives and in planning and scheduling assessment
activities, and all FY04 BNL required assessments relevant to EENS were
acknowledged in the RO SA Plan.

1.  **The scopes of planned assessment activities are comprehensive,
    have a strong technical basis, and are balanced with work
    activities.**

Assessment activities are prioritized based on alignment with BNL and
organizational strategic plans, results of previous assessments,
business and operational risk, and available resources. EENS Departments
align Strategic Plans with the BNL Institutional Plan and Laboratory
Strategic Initiatives. EENS Strategic Plans are also aligned with DOE
and other sponsors' plans where appropriate. **(+)** EENS held a
Strategic Planning Retreat in January 2004 at which long range planning
was conducted. Initiatives were also proposed and discussed at the
Retreat. EENS has instituted planning committees at both Department and
Directorate levels to provide both input and feedback for these planning
processes. RO planned activities align with research plans, operational
risk, and SBMS requirements. BO activities align with research plans and
business risk as well as financial management requirements.

The RO SA Plan design is based on the IAP (Baldrige) framework Criteria
4.0 and 6.0. **(-)** The RO SA Plan does not explicitly acknowledge
Criterion 3.0, Customer Focus and Satisfaction (Customer Value). The RO
SA Plan explicitly documents linkage to BNL Critical Outcome 3.0
objectives. EENS Strategic Plans, Strategic Initiatives, and reporting
to the Integrated Information Management System (IIMS) evidence
appropriate linkage to BNL Critical Outcome 1.0 objectives.

**(-)** There is no Directorate-level SA Plan or roadmap linking other
documents to SA; there are no finalized Department-level SA Plans. BO
has no SA Plan. That said, the EENS SA program is comprehensive in that
it covers the full scope of the Directorate's research, operational,
business, and financial activities. Directorate and Department Strategic
Plans and Strategic Initiatives cover the "scientific basis" of research
activities, contain "near-term" and "mid-term objectives," establish
"marketing strategy," and identify "metrics" ("measures of success").
**(+)** The RO SA Plan comprehensively covers ES&H as well as
operational elements including environmental management, work planning
and control, worker safety and health, environmental compliance, and
radiation control. **(+)** The Annual Fiscal Year-End Analyses
\[report\] prepared by BO reflects comprehensive coverage of the
Directorate's business/financial activities including human resources,
cost and balances for each research project by Department and the CDIC,
indirect programs, organizational burden, and staff recharge and
subcontracting. **(+)** The EENS strategic planning process explicitly
incorporates the need for ESH and infrastructure upgrades as implemented
through the Project, Planning, Programming and Budgeting Process (3PBP).
The RO SA Plan explicitly includes submitting "Activity Data Sheets
(ADSs) for infrastructure improvements/project-related initiatives."
This was noted previously in the Independent Assessment of 3PBP (IO
03-13).

EENS Strategic Plans reflect appropriate consideration of BNL and DOE
expectations as communicated through BNL Critical Outcome 1.0, DOE's
Annual Evaluation of BNL, DOE Program Reviews, DOE Strategic Plans, DOE
Field Work Proposals (FWPs), NNSA and Work for Others (WFOs) agreements,
and frequent meetings by EENS managers with DOE and other sponsors. The
RO SA Plan reflects appropriate attention to SBMS (hence DOE)
requirements. **(-)** The RO SA Plan does not incorporate customer
satisfaction measures, although the RO Manager meets periodically with
other EENS managers (customers). The Annual Fiscal Year End report
content/format is based on input from EENS customers.

The effectiveness of previously completed corrective actions is
appropriately considered during planning. EENS Strategic Plans reflect
consideration of completed actions in generating updated strategies. The
RO SA Plan explicitly incorporates approaches that are indicative of
effectiveness of previously completed actions by including "follow up on
actions delineated in the last reporting period," for example.

Results from external assessments are incorporated into EENS's
self-assessment and annual self-evaluation processes. Examples include
program reviews, DOE's annual evaluation of BNL (Critical Outcome 1.0),
BNL and DOE assessments of ES&H, and audits of financial controls.
During Tier I inspections, it was confirmed that EENS has responded to,
and continues to address, items noted during the recent OSHA inspection
of BNL.

External lessons learned are incorporated into the EENS self-assessment
process. Lessons are incorporated through involvement of peers in
program reviews and subject matter experts (SMEs) in Tier I inspections,
for example. EENS personnel subscribe to the BNL Lessons Learned
service. EENS makes effective use of external peer/expert reviews to
incorporate external experience into scientific, operational, and
safety-related aspects of the Directorate. Additionally, personnel also
participate on external committees, are active members of
associations/collaborations, and attend conferences and workshops.

2.  **The methods for conducting key scheduled assessments are defined
    and are commensurate with types of assessments planned and
    performance information desired.**

Assessment activities are assigned to qualified personnel including
appropriate use of SMEs. EENS uses expert, qualified peers to conduct
research program reviews. RO assessments routinely involve SMEs such as
the Environmental Compliance Representative (ECR) and Facility Support
Representative. The IO Reviewer observed the participation of SMEs on a
Tier I inspection. RO personnel indicated disappointment that industrial
hygiene (IH) SMEs did not routinely attend Tier I inspections though
invited. **(+)** RO has established a performance measure to obtain more
IH SME attendance in FY04.

Managers effectively communicate their expectations such that employees
understand their roles and responsibilities for self-assessment.
Expectations are communicated routinely in R2A2s and performance goals
for managers and staff. The RO SA Plan explicitly lists "tasks" and
assigns "responsibility" for each. EENS Strategic Plans and Strategic
Initiatives identify "responsible personnel" for specific actions.

Managers ensure that independent assessments and peer reviews are
considered and incorporated into the self-assessment process, as
appropriate. Independent assessments conducted on behalf of management
system stewards (Environmental Management System \[EMS\], Work Planning
and Control, Worker Safety and Health, and Radiological Control) are
explicitly included in the RO SA Plan. EENS makes extensive use of peer
and management reviews of proposals and documents submitted for
publication. As mentioned previously, EENS uses external committees
(advisory, visiting, BSA S&T Committee, DOE Review, etc.) to provide
peer review. Tier I inspection teams are made up largely of independent
inspectors.

The assessment approach uses an appropriate mix of information sources.
EENS uses peer reviews, research program reviews (preparation for
presentations including sponsor/peer review), Tier I inspections
(tracking/trending), SBMS required assessments (audits, questionnaires,
document reviews, readiness reviews, in-field surveys, emergency
drills), and business/financial assessments (cost tracking, subcontract
monitoring, purchase card audits, property management), for example.

Tier I inspections are conducted quarterly in laboratory spaces and
annually in office spaces. Inspections take an hour or two to complete.
Researchers are involved for some fraction of the time it takes to
complete an inspection and seem willing to be involved. During the Tier
I observed, fruitful discussions were held between inspectors and space
occupants. There is little or no adverse impact to work activity.

3.  **Management and staff involvement commensurate with their
    responsibilities is evident.**

Planning involves managers and staff members as appropriate to ensure
that all aspects of the organization's operations are evaluated. The
EENS Strategic Planning Retreat involved the ALD, Special Assistant to
the ALD, Department Chairs, Division Heads, principal investigators, RO
manager and deputy, and the BO manager. Departmental strategic planning
involves group leaders and research staff.

The RO SA Plan was reviewed by the RO manager and approved by the ALD.
**(-)** The RO SA Plan is dated 4/01/04. The date of April 1 does not
meet the requirement (from Integrated Assessment subject area) that the
"update of the organization\'s self-assessment program for the current
year must be completed by the end of the first quarter (December 31) of
each fiscal year." **(-)** Further, there were no other SA Plans issued
in EENS.

4.  **Key supporting organizational processes (tracking/trending
    systems, causal analysis, critiques, etc.) and tools are
    developed.**

The organization has effective processes for analyzing and trending the
results of assessment activities. RO personnel trend findings from Tier
I inspections. RO personnel also participate in critiques and/or causal
analyses to analyze results of assessment activities and/or
nonconformances. Tier I inspections and other assessment activities have
identified issues with management of waste in Satellite Accumulation
Areas, as also noted prior to a recent Resource Conservation and
Recovery Act (RCRA) inspection. Causal analysis will be conducted in
concert with other "small science" organizations.

RO personnel also analyze results from other assessment activities and
document this analysis in a year-end report. BO provides
business/financial status summary in a year-end report. The BO year-end
report for FY03 is the initial effort; hence, there is no trending
versus previous years. EENS research performance is analyzed in
management meetings; analysis is evident in Strategic Plans, **(-)**
though not necessarily evident in EENS's contributions to the year-end
BNL Self-Evaluation.

The organization has effective processes for tracking corrective,
preventive, and improvement actions to completion; although, there is no
unified process for tracking actions. EENS personnel do not use the
Family Assessment Tracking Systems (FATS) available to them. RO tracks
actions resulting from Tier I inspections and documents action status in
Tier I reports. Other actions are tracked by responsible managers using
formal/informal means. Institutional-level actions are tracked in ATS;
some actions are tracked in other organizations' (e.g., Environmental
and Waste Management Services \[EWMS\]) FATS.

Records are maintained that demonstrate effective planning, timely
completion of assessments, and analysis of results. The RO SA Plan is
maintained as a controlled document. RO provided several examples of
Tier I inspection reports, nonconformance reports, and the Worker Safety
and Health Required Assessment Aid (FY03) demonstrating timely
completion of assessments. The Annual Fiscal Year-End Analyses documents
review of business/financial functions within EENS. Strategic Plans also
reflect analysis of results from previous years. Contributions by EENS
managers are incorporated in BNL's Year-end Self-Evaluation serving as
records of timely submittal.

Some available information systems are used to effectively support
assessment activities. RO uses a database to trend results from Tier I
inspections. BO personnel use PeopleSoft to analyze and report financial
information periodically. Research performance results are accumulated
using IIMS. EENS websites contain internal procedures. Institutional ATS
(external and/or independent assessments per ESH 1.2.1) and EWMS FATS
are used to track some EENS/BNL assessment activities as discussed
previously.

5.  **Organizational plans/procedures address regulatory and/or SBMS
    drivers for assessment activities.**

Assessment activities are based on and linked to the organization's
performance objectives and measures as well as the BNL Critical Outcomes
and associated Performance Objectives/Measures. RO assessment activities
(tasks) are clearly linked to the IAP Assessment Criteria, Critical
Outcome 3.0, and EENS Objectives. Strategic Plans provide linkage to BNL
Critical Outcome 1.0 and the BNL Institutional Plan; "success measures"
are linked with objectives in the Strategic Plans.

All applicable "Required Assessments" are identified in the RO SA Plan
demonstrating appropriate consideration of integration of SBMS
Management System requirements. BO personnel could not recall whether or
not FY03 required assessment aids (Acquisition Management) were
completed. Acquisition Management contacts could not verify whether or
not FY03 assessment aids had been submitted.

### Criterion 2: Deployment of Self-Assessment

Assessment activities are conducted as scheduled. Schedules may be
adjusted based on changing priorities as determined by organizational
managers. Results of assessment activities are documented and
communicated to responsible EENS managers for review and analysis. Key
assessment results are regularly discussed and analyzed at
management/staff meetings. Managers evaluate assessment results to
identify both strengths and opportunities for improvement. Necessary
improvement actions are identified and communicated to responsible
managers and stakeholders.

1.  **Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.**

The RO SA Plan contains the only documented schedule for assessment
activities, and activities have been completed as scheduled or
rescheduled when appropriate. The IA&O Reviewer observed a quarterly
Tier I inspection that was conducted as scheduled. The annual EMS
Management Review was completed in November 2003; the IA&O Reviewer
attended. The IA&O Reviewer also attended the Strategic Planning Retreat
that was conducted in January 2004. FWPs were prepared, reviewed, and
submitted per schedule. Mid-year and year-end reporting for research
(Critical Outcome 1.0) has been accomplished according to BNL's
schedule. BO provides periodic (monthly/quarterly) financial/business
results per customer requirements. **(+)** BO year-end analyses were
completed in a very timely fashion -- dated October 14, 2003.

The RO SA Plan, which is managed as a controlled document, will be
updated to reflect changes in scope. The RO SA Plan notes Revision 5 to
indicate that the SA Plan has been revised five times since original
issuance in 2000. The SA Plan has not been updated since initial
issuance in FY04 since there have been no significant changes to
incorporate. Scope and schedule of planned assessments are adjusted when
appropriate based on significant performance information. The RO SA Plan
provides an example of adjustment in responding to RCRA concerns
identified during Tier I inspections. EENS Strategic Plans are updated
at least annually. Strategic Plans are viewed as living documents in the
sense that they do not become final in any year.

Customer feedback is solicited during performance appraisals. DOE
feedback is provided in the course of program reviews. EENS managers
solicit customer feedback during meetings held with DOE and other
research sponsors, although this activity is not formally documented in
an assessment plan. **(-)** The RO SA plan does not include soliciting
customer feedback as part of planned assessment activities.

2.  **Assessments are documented and communicated as planned.**

Assessment activities are documented and results are provided to
responsible managers. RO provides reports to affected personnel for Tier
I inspection results, for example. RO also reports to EENS management at
year-end. BO provides the Annual Fiscal Year-End Analyses to EENS
managers. EENS periodically reports status of Strategic Initiatives to
the Integrated Planning Office manager. Mid-year and year-end reports
addressing BNL Critical Outcome 1.0 are submitted to the Integrated
Information Management System (IIMS).

The assessment schedule is managed to ensure information is available to
support timely management decisions. Assessment activities addressing
research program performance are timed to permit submittal to IIMS per
BNL's schedule for submission of the annual Science and Technology Year
End Self-Evaluation \[S&T Report\] to DOE. The FY03 S&T Report is dated
October 8, 2003. The assessment schedule as documented in the RO SA Plan
is timed to conclude other activities before the end of the year. FWPs
were submitted in February to allow for timely preparation of budgets
for FY05.

3.  **Assessment results (internal and external) are evaluated/analyzed
    to a degree commensurate with the type of assessment. Strengths are
    identified. Corrective/improvement actions are identified,
    prioritized, assigned to specific owners, and assigned due dates.
    All conditions are tracked to closure. Actions are validated as
    improving performance.**

EENS has an effective process for identifying opportunities for
improvement. The RO year-end report specifically identifies
opportunities for improvement. Tier I reports list findings. **(+)**
Identified opportunities for improvement include Project Planning,
Programming, Budgeting & Activity Data Sheet Process (3PBP) submittals
to upgrade laboratory spaces. Strategic Plans and The BNL Year End
Self-Evaluation identify opportunities for improvement in research.
**(-)** The BNL Year End Self-Evaluation is not sufficiently analytical
or self-critical in identifying areas for improvement when contrasted
with DOE's evaluation of BNL's performance.

Improvements are prioritized in accordance with business and operational
risks and cost benefit. EENS Strategic Plans prioritize potential new
research projects in terms of amount of proposed funding and
appropriateness of research. 3PBP is used to address identified needs
for ES&H and infrastructure improvements in accordance with risk.

Improvement actions are developed with consideration of causal factors
as commensurate with the level of hazard/risk. RO analyzes identified
trends in ES&H for causal factors. **(+)** The RO SA Plan states,
"Encourage improvement activities by initiating and organizing a
facilitated critique/root cause analysis of RCRA findings with
participation of BNL small science Departments." Analysis of causal
factors is evident in strategic planning for research though not
explicitly documented. Examples include need for strategic hires or need
for upgrades in laboratory spaces.

Change control for action due dates reflects consideration to balance
priorities. Change control for institutional-level actions is managed in
accordance with ESH 1.2.1. Action due dates within EENS are controlled
by action owners and their managers.

Improvements are validated, as commensurate with the level of
hazard/risk, to ensure desired outcomes are achieved including
prevention of recurrence of adverse events. Tier I inspections include
follow-up of previous findings. Some Tier I findings are addressed at
the time of the inspection. Trending of Tier I findings identifies
recurring events. The RO approach for Radiation Control specifically
includes appraising the effectiveness of Radiological Awareness Reports
(RARs) and Occurrence Reporting and Processing System (ORPS) corrective
actions. External assessments (e.g., program reviews, annual DOE
evaluation) provide validation of BNL actions by acknowledging
improvement and/or lack of recurrence.

4.  **Evidence of timely self-identification of issues exists.
    Significant issues are brought to the attention of management and
    disclosed to regulatory/oversight agencies in accordance with
    contractual obligations.**

Results are analyzed to identify potential noncompliances with nuclear
safety rules (Price-Anderson Amendments Act \[PAAA\]). There is no
specific mention of PAAA in the RO SA Plan; however, the RO Manager is a
member of the PAAA Working Group who is highly cognizant of PAAA issues.
In the RO SA Plan, the "Objective: Activities in the EENS Directorate
are expected to be conducted in compliance with applicable ES&H Laws and
Regulations" features a comprehensive approach for Radiation Control.
Facility Support personnel write RARs, which are reviewed by PAAA
Coordinator. Occurrence Reports are also reviewed by the PAAA
Coordinator. Facility Support Representatives also conduct quarterly
self-assessments (as required by the Radiological Control Division's SA
Plan) in the Life Sciences Directorate.

Results are analyzed to determine potential impacts on SBMS documents.
SBMS required assessments specifically address either Management Systems
or individual Subject Areas. As mentioned previously, RO activities
specifically address SBMS Management Systems including EMS, Work
Planning and Control, Worker Safety and Health, and Radiation Control.
EENS is involved in the improvement of the Tier I inspection process. RO
tracks the "number of BNL Management Systems development/improvement
initiatives participated \[in\] by staff." Feedback from EENS was used
to identify areas for improvement in IAP as reported in Section 3.3 of
this report.

Results of assessment activities have been appropriately considered for
publication in the BNL Lessons Learned Program. EENS personnel have
authored BNL Lessons Learned and prepared lessons learned narratives in
ORPS reports. The RO SA Plan documents a task to "submit lessons
learned/success stories to the \[Pollution Prevention\] Council."

5.  **Management involvement is evident.**

Managers participate in planning and conducting assessment activities.
Per the RO SA Plan, the RO Manager and Deputy Manager are involved in
planning and conducting assessment activities. The BO Manager is
involved in planning/conducting activities related to business/financial
management including development and approval of the Year-End Analyses.
As mentioned previously, the ALD, Chairs, and Division Heads attended
the EENS Strategic Planning Retreat. Group Leaders, Division Heads,
Department Chairs, and the ALD are involved in developing Strategic
Plans and evaluating performance at year end.

EENS managers conduct informal walkthroughs that may involve observing
work in progress. Chairs acknowledged walkthroughs of buildings in which
they have personnel. These walkthroughs are not routinely documented.

EENS managers participate in the analysis and evaluation of assessment
results as evidenced in the RO CY 2003 Self-Evaluation, the FY 2003 Year
End Fiscal Analyses, and the EENS sections in The BNL Year End
Self-Evaluation. Analysis and evaluation are also evident in EENS
Strategic Plans and Strategic Initiatives. Managers also conduct annual
Performance Appraisals of their direct reports.

Managers verify that needed improvements are implemented. Tier I
inspections include following up on actions resulting from previous
inspections to verify that actions have been implemented. Tier I
inspection results are monitored (trended) for recurrences as a means of
ensuring that implemented actions have been effective. EENS managers'
year-end evaluations of individual and organizational performance verify
that improvements in research have been effectively implemented.

### Criterion 3: Results from Self-Assessment

The EENS Directorate provided timely input to the FY03 BNL Year End
Self-Evaluation Report as in previous years summarizing performance
results for Critical Outcome 1.0 for Departments and CDIC. Critical
Outcome performance reporting was accomplished via computerized
submittals to IIMS. The Directorate also documented performance in the
EENS Environmental Management Review Meeting Minutes. The RO Office
Manager provided the RO Self-Evaluation to the ALD. Reports of
performance described improvements achieved and established a basis for
planning. Results are communicated to BNL senior management. There is
appropriate linkage between the EENS Strategic Plans and Strategic
Initiatives and BNL's Institutional Plan.

1.  **Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations, such as mission
    achievement and retention/expansion of core competencies.**

Progress towards achievement of near-term and long-term goals and
objectives is evident. EENS Strategic Plans and the BNL Year End
Self-Evaluation document progress. The BNL FY03 Year-end Report
documents (Section 1.3.6) achievement of the scientific and technical
objectives contained in EENS Strategic Plans and Strategic Initiatives.
Presentations and discussions at the EENS Strategic Planning Retreat
also highlighted these and other achievements. Strategic hires of
scientific personnel are evident in EENS Departments. The RO Self
Evaluation also highlights progress toward achievement of goals.

The BNL FY03 Year-end Report (Section 1.3.6) and RO Self Evaluation
describe how conduct of operations/research has been enhanced in the
EENS Directorate. As an example, four of the Free-Air Carbon Dioxide
Enrichment (FACE) facilities became a single official DOE Distributed
User Facility. An EENS scientist was named the FACE Facility
Coordinator. EENS assists with design of experiments, provides
engineering support, participates in research activities, promotes FACE
at scientific meetings, maintains the **(+)** FACE website (including
quality control and performance information), and, in general, provides
a body of knowledge/expertise regarding FACE. EENS also reported the
design, construction and operation of the Radiation Detector Testing and
Evaluations Center (RADTEC) that is expected to result in a multiyear
test-bed program. Also, appropriately selected 3PBP projects, linked to
research needs, have resulted in enhancements of research workspaces.
Improvements in ES&H are documented in the RO Self Evaluation.

Evidence exists that customers and other stakeholders value the products
and services of the EENS Directorate. Much of this evidence is in the
form of renewed and newly awarded research projects. There is increasing
demand to use FACE and RADTEC facilities as well as increased demand for
information form the National Nuclear Data Center (NNDC). EENS's Office
of Science (SC) research portfolio received Excellent/Outstanding
ratings in SC's FY 2003 BNL Evaluation; although, as mentioned
previously, SC's review is more critical than BNL's self-evaluation. RO
personnel involvement in SBMS developmental activities has received
favorable comment from other Departments at BNL. During interviews, EENS
managers spoke highly of support provided by RO and BO personnel.

EENS organizations provided specific examples of improvements resulting
from the self-assessment process including improvements to the
self-assessment process itself. FACE and RADTEC serve as two examples
where self-assessment (planning, setting objectives, measuring
performance, and analyzing results) resulted in improvements. An example
of improvement to the SA process is the inclusion of RO and BO managers
in the Strategic Planning Retreat. The EENS Tier I inspection process
serves as another example where a self-assessment activity has been
improved by encouraging both management and SME participation and by
shortening turn-around time on reports of findings.

2.  **Evidence exists that there is an appropriate connection between
    results of organizational self-evaluation and development of
    strategic/institutional plans.**

EENS Strategic Plans are modified as necessary based on evaluation of
assessment results and evidence consideration of DOE Strategic Plans and
the BNL Institutional Plan. EENS strategic plans are living documents
that, per the ALD, are not expected to become "final." They are adjusted
as need arises based on research results, budgetary concerns, etc. EENS
Strategic Plans are prepared in consideration of, and clearly reflect,
DOE (and other) strategic plans. EENS Strategic Plans and Strategic
Initiatives are used as input to the BNL Institutional Plan.

Improvement actions are either acted upon at the
organization/Directorate level or communicated to BNL senior management
to be included in the Laboratory's improvement agenda, as appropriate.
EENS Strategic Plans and Strategic Initiatives incorporate actions;
these actions may be taken at local, Department, or Directorate levels
and may also be communicated to BNL senior management in the BNL
year-end report and/or Laboratory Strategic Initiatives. Additionally,
evaluation results were communicated to the Manager of the Environmental
and Waste Management Services Division who attended the EENS EMS
Management Review.

In addition to the annual BNL self-evaluation process, EENS demonstrated
some responsiveness to other independent or external assessments as
exemplified via improvements addressing the Areas for Improvement and
the Recommendations included in **IO Report SA 01-01**:

-   **The EENS Self-Assessment Plan for FY 2001 was not finalized at the
    time of this evaluation.** This statement referred to the overall SA
    Plan for the EENS Directorate. For FY04, the RO SA Plan was
    completed though not by 12/31/03 as required in the Integrated
    Assessment subject area. There is no overarching SA Plan for the
    EENS Directorate.

-   **Both EENS and OIP identified areas for improvement in their FY
    2001 annual self-evaluations that have not yet been fully
    addressed.** For EENS this spoke to the RO Office Manager having
    identified a need to improve communication of Tier I results and
    trending of Tier I results. As of this FY04 IA&O Review, timeliness
    of communication of Tier I results has been improved and trending
    has been implemented.

## Conclusions

The EENS Directorate evidences improvement in the organizational
Self-Assessment Program since FY01 over what was already recognized as
an effective program. The SA Program is comprehensive in scope, although
only RO's SA Program is formally documented. Approaches to
self-assessment activities include more than just traditional audits and
represent a performance-based concept. Roles and responsibilities for
self assessment are defined. Processes for documenting results of
activities, analyzing results, identifying opportunities for
improvement, and tracking actions are in place. The RO SA Plan, EENS
Strategic Plans, and EENS Strategic Initiatives document an annual cycle
of continuous improvement in effectiveness and efficiency, represent a
forward-looking approach, and provide linkage with the BNL Institutional
Plan in planning and achievement of longer-term goals and objectives.

### Strengths 

-   EENS Directorate managers interviewed uniformly articulated
    commitment to improving organizational and individual performance as
    reflected in the research, services, and/or products provided by the
    Directorate. As mentioned previously, specific improvements are
    documented in the FY03 BNL Year End Self-evaluation, presentations
    for the Strategic Planning Retreat, the RO Self Evaluation, and the
    EMS Management Review Minutes. Notably, managers indicated a desire
    for further improvement even in areas that were highly rated by
    customers/stakeholders.

```{=html}
<!-- -->
```
-   The EENS Strategic Planning Retreat provided an effective way of
    identifying long-range goals and objectives for the Directorate. The
    BNL Laboratory Agenda was featured as input to the planning
    discussions ensuring linkage between EENS and BNL goals/objectives.
    Inclusion of RO and BO managers in the retreat facilitated the
    appropriate consideration of ES&H and financial elements in
    planning. Strategic Initiatives were also proposed and discussed
    during the Retreat.

-   The comprehensive content of the RO SA Plan provides an exemplary
    model that goes beyond the requirements of the Integrated Assessment
    subject area. The Plan documents linkage to Critical Outcomes and
    the IAP framework. Approaches specifically address elements of key
    SBMS Management Systems. A comprehensive set of metrics is appended
    to each objective.

-   The RO Self Evaluation provides a concise, comprehensive,
    self-critical analysis of performance versus the RO SA Plan
    for 2003. This document also provides an exemplary model of an
    organizational self evaluation in that it identifies strengths and
    areas for improvement and provides recommended actions.

-   The FY 2003 Annual Fiscal Year End Analyses prepared by the BO
    Office provides a comprehensive summary of business/financial
    performance of the Directorate and Departments. The report date of
    October 14, 2003, evidences a very timely report of this information
    to EENS management.

-   Infrastructure objectives are clearly considered as part of
    strategic planning in EENS. The need for upgrades of laboratory
    spaces is identified as part of the EENS planning process. Status of
    3PBP proposals/projects is discussed at EENS management meetings.
    The RO SA Plan specifically includes "submittal of Activity Data
    Sheets for infrastructure improvements/project-related initiatives."

-   The Free Air Carbon Dioxide Enrichment (FACE) website contains and
    communicates meaningful performance information in an effective and
    efficient manner. The website contains Performance Reports
    ("presenting a near real time analysis of the conditions and
    performance of FACE sites") and Quality Assurance information
    (Design Considerations, Science Management, Operations Guidelines,
    QA Procedures) useful to both the facility manager and facility
    users.

-   EENS articulated a desire to increase the participation of SMEs,
    particularly those with industrial hygiene expertise, on Tier I
    inspections and readiness reviews. Notably, RO tracks SME
    participation (e.g., % of Tier I inspections with IH participation)
    as a performance metric.

-   Trending/analysis of Tier I results versus RCRA requirements led to
    a proactive approach to resolving identified issues. The intent to
    work in concert with other "small science" organizations represents
    an efficient approach for the Directorate and for BNL. EENS
    leadership in this effort was acknowledged by personnel in other BNL
    organizations.

### Areas for Improvement 

-   While it is clear that assessment activities covering aspects of
    research and business/financial management in EENS are occurring,
    these activities (and corresponding objectives) are not documented
    in Directorate or Department SA Plans. No EENS SA Plans were
    finalized by

December 31, 2003.

> **Recommendation:** The ALD for EENS should ensure that, at a minimum,
> an overarching document (EENS SA Plan) is prepared that documents
> connection (roadmap) to EENS Strategic Plans and Strategic Initiatives
> and to the business/financial objectives of EENS. The RO SA Plan
> serves an effective starting point for developing the EENS SA Plan.
> Research and business objectives could be appended to the RO SA Plan
> in the appropriate format as used by RO. Such a document should prove
> useful to EENS managers in determining resource needs for SA and
> better enabling reallocation of resources when necessary. The ALD
> should also ensure that annual updates to the EENS SA Program are
> documented before December 31 each year.

-   The RO SA Plan, Rev. 5, does not include objectives or approaches
    addressing customer satisfaction.

> **Recommendation:** The RO Manager should develop customer
> satisfaction objectives and approaches and document these in the
> updated RO SA Plan.

-   The EENS year-end submittal to the BNL Annual Self Evaluation
    reports separate individual accomplishments without providing an
    analytical evaluation of research performance. The year-end
    submittal is not sufficiently self-critical in that EENS did not
    identify and report areas for improvement that DOE identified and
    reported in their Annual Evaluation of BNL.

> **Recommendation:** The ALD for EENS should ensure that
> analysis/evaluation of research performance is included either in the
> EENS submittal to IIMS or in a separate document providing a self
> evaluation of EENS. This submittal/document should identify both
> performance strengths and areas for improvement.

### IAP Management System Programmatic Issues

It is acknowledged that some of the issues discussed below are known to
BNL management and the IAP Management System Steward and Point of
Contact, and that efforts may be underway to address these issues. These
issues are noted here for emphasis.

Interviewees expressed concern about the timing (January 2004) of the
most recent revision to the Integrated Assessment Subject Area
incorporating new required assessments and the need for organizations to
update their SA Programs/Plans shortly after initial issuance/approval
as required by December 31 each year. Interviewees also expressed
concern about the publication dates for BNL Critical Outcomes (Appendix
B) and Environmental Objectives, which both occurred after

December 31, 2003. This flawed timing creates the appearance of
organizations' being out of conformance to the Integrated Assessment
subject area while awaiting the publication of key planning elements
vital to their SA Plans.

An interviewee requested that BNL management provide better guidance as
to what constitutes an acceptable SA Plan. The Integrated Assessment
subject area states, "Document the self-assessment **program** in
conformance with the requirements specified on the Self-Assessment
**Program** Requirements exhibit." The exhibit in turn states, "All of
the requirements listed ... do not need to be included in a single
document." The subject area also states that the "update of the
organization\'s self-assessment **program** for the current year must be
completed by the end of the first quarter (December 31) of each fiscal
year." The Integrated Assessment subject area does not specifically
refer to any document known as a Self-Assessment **Plan**. It seems that
BNL management expects organizations to have SA Plans without
specifically requiring that SA Plans exist.

Also noted during the course of this review, was the fact that a large
portion of research performed in the EENS Directorate is not part of the
basis for DOE's Annual Evaluation of BNL. Only research sponsored by the
Office of Science is used by DOE. BNL's Self Evaluation incorporates all
DOE, NNSA, and research conducted for others to rate performance.
Further DOE rates performance for Critical Outcome 1.0 based on an
accumulation of project by project scores whereas BNL's accumulation
seems to be on a Department by Department basis. Neither DOE nor BNL
defines how these ratings are accumulated. The result is an apparent
"apples to oranges" comparison affecting 60% of BNL's award fee.

[]{#__RefHeading___Toc68064789 .anchor}Exhibit 1 - Documents Reviewed

1.  Energy, Environment, and National Security CY 2004 Self-Assessment
    Plan: Research Operations Office, Rev. No. 5 (4/01/04)

2.  Energy, Environment, and National Security CY 2003 Self-Evaluation:
    Research Operations Office (12/31/03)

3.  Memorandum from J. Boccio to R. James, Research Operations Office
    Self Evaluation, April 1, 2004

4.  Energy, Environment, and National Security Directorate Annual Fiscal
    Year-End Analyses (October 14, 2003)

5.  Memorandum from J. Indusi to R. James, Draft Self-Assessment Plan
    for NNSD (April 14, 2004)

6.  EENS Strategic Planning Retreat Agenda, January 26, 2004

7.  EENS Strategic Planning Retreat Initiative \[Form\]

8.  Report of the Review Committee for the Environmental Sciences
    Department, Brookhaven National Laboratory, July 14 -- 16, 2002

9.  Report of the Visiting Committee's Review of Brookhaven National
    Laboratory's Energy Sciences and Technology Department, July 29 --
    31, 2002 (Final Review Draft 8/29/02)

10. Memorandum from R. B. James to M. Furey, et al., Strategic
    Initiatives/New Hire Funding Approvals, January 31, 2003

11. \[Presentation:\] Energy, Environment and National Security (EENS)
    Strategic Planning, S&T Steering Committee, March 9, 2002

12. Memorandum from P. Moskowitz to R. James, et al., Tactics for CT/HS
    \[Counterterrorism/Homeland Security\] Initiative, December 17, 2001

13. Advanced Sensors Initiative, Nonproliferation and National Security
    Department, nd

14. Initiative Plan -- Advanced Fuels, Energy Sciences and Technology
    Department, December 2001

15. Nuclear Energy (DOE/NRC/Industry) Initiatives, nd

16. Strategic Planning for TAP \[Tropospheric Aerosol Program\] (Draft
    12/12/01)

17. Strategic Planning for TERF \[Terrestrial Ecosystem Research
    Facility\] (Draft 12/12/01)

18. Strategic Planning for GTL \[Genome to Life\], A Partnership with
    Biology Department (Draft 12/12/01)

19. Strategic Planning for EnviroSuite, A NSLS Consortium (Draft
    12/12/01)

20. Letter from N. Torres \[DOE/BAO\] to K. Fox, Notice of GAO Visit to
    BNL (12/09/03)

21. E-Mail from J. Nash \[GAO\] to N. Torres, Upcoming GAO Visit to
    Brookhaven (12/09/03)

22. Letter from L. Williams \[GAO\] to S. Abraham, Secretary DOE, July
    25, 2003

23. Letter from G. Aloise \[GAO\] to P. Moskowitz, May 8, 2003

24. NNS Department Suggestion Site \[webpage\] (printed 3/31/04)

25. \[Factsheet:\] Brookhaven's Radiation Detector Testing and
    Evaluation Center (RADTEC) (3/03)

26. \[Factsheet:\] TEMASU: the Terrestrial Magnetic Surveyor, October
    2002

27. \[Factsheet:\] Brookhaven's Regional Counterterror Activities (3/03)

28. Performance Reports \[www.face.bnl.gov/FACEDailyReports\] (printed
    4/08/04)

29. QA Issues at FACE Research Facilities
    \[www.face.bnl.gov/quality.htm\] (printed 4/08/04)

30. The Urban Atmospheric Observatory \[www.uao.bnl.gov\] (Last edit: 16
    Feb 2004)

31. \[Energy Sciences and Technology Department\] Suggestions
    \[webpage\]

32. \[Factsheet:\] Human Factors Research and Applications, nd

33. \[Presentation:\] Nuclear Analysis Capability, Brookhaven National
    laboratory, Energy Sciences and Technology Department, January 2004

34. Project Outlines, Center for Data Intensive Computing
    \[www.bnl.gov/cdic/Sci_Projects\] (printed 4/16/04)

35. E-Mail from C. Sauter to Distribution, Tier I Schedule for Quarter
    2/2004

36. Memorandum from J. Boccio to T. Roberts/Y. Celebi, Results of Tier I
    Inspection, March 12, 2004

37. EENS Tier I Inspection Report, Building 0815C, February 5, 2004

38. \[EENS\] Non-Conformance Form, Incident No. NE-04-01 (2/09/04)

39. \[EENS\] Non-Conformance Form, Incident No. NE-04-02 (2/10/04)

40. \[Presentation:\] EENS Directorate EMS Management Review, November
    12, 2003

41. Minutes: EENS Environmental Management Review Meeting, November 12,
    2003

42. E-Mail from P. Carr to R. Lebel, Facilitation of Critiques/Root
    Cause Analysis, January 7, 2004

43. Example Results from Trending of Tier I Findings

44. Laboratory/Storage Area Safety Self Inspection Checklist

45. EENS Business Operating Procedures \[webpage\]

[]{#__RefHeading___Toc68064790 .anchor}Exhibit 2 -- Personnel
Interviewed

1.  R. James, ALD for Energy, Environment and National Security

2.  M. Furey, Special Assistant to the ALD for EENS

3.  J. Boccio, Manager, Research Operations Office

4.  P. Carr, Deputy Manager, Research Operations Office

5.  A. Fridae, Manager, Business Operations Office

6.  W. Horak, Chair, Energy Sciences and Technology Department

7.  J. Indusi, Chair, Nonproliferation and National Security Department

8.  C. Wirick, Chair, Environmental Sciences Department

9.  J. Davenport, Associate Director, Center for Data Intensive
    Computing

10. G. Hendrey, Head, Earth Systems Science Division, Environmental
    Sciences Department

[]{#__RefHeading___Toc68064791 .anchor}Exhibit 3 - Documents Referenced

1.  SBMS Management System Description: Integrated Assessment Program,
    Issue Date: July 1999

2.  BNL Independent Oversight Office FY 2004 Program Plan: Review of
    Organizational Self-Assessment Programs (December 30, 2003)

3.  U.S. Department of Energy Contract with Brookhaven Science
    Associates, DE-AC02-98CH10886

4.  Appendix B (to DE-AC02-98CH10886), Critical Outcomes, Objectives,
    and Performance Measures FY 2004 Brookhaven National Laboratory
    (Effective: 03/16/04)

5.  Year End Self-Evaluation - Brookhaven National Laboratory -- Science
    and Technology - Fiscal Year 2003 (October 8, 2003)

6.  Year End Self-Evaluation - Brookhaven National Laboratory - Fiscal
    Year 2003 (December 19, 2003)

7.  U.S. Department of Energy Fiscal Year 2003 Annual Evaluation of
    Brookhaven National Laboratory (April 19, 2004)

8.  Brookhaven National Laboratory Draft Institutional Plan FY 2004 --
    FY 2008, September 2003

9.  SBMS Subject Area: Integrated Assessment, Effective Date: January
    2004

10. SBMS ESH Standard 1.2.1, Corrective Action Management and Tracking
    for External and Internal Assessments, Rev. 5, Effective Date:
    December 2002

11. Brookhaven National Laboratory Independent Oversight Report SA
    01-01, Evaluation of Self-Assessment Programs of the Energy,
    Environment and National Security Directorate and the Office of
    Intellectual Property and Industrial Partnerships (March 21, 2001)

12. Brookhaven National Laboratory Independent Oversight Report IO
    03-13, Project, Planning, Programming, and Budgeting Process (3PBP)
    and Activity Data Sheet (ADS) Process (March 10, 2004)