Brookhaven Science Associates

**Brookhaven National Laboratory**

**Independent Oversight Report**

**SA 04-01**

**Review of the Self-Assessment Program**

**of the**

**Life Sciences Directorate**

**April 27, 2004**

**Prepared by:**

**John Usher, Reviewer Date**

> **Independent Oversight**

**Approved by:**

**Robert McNair, Manager Date**

**Independent Oversight**

Contents

1.0 Introduction [1](#introduction)

1.1 Purpose and Scope [1](#purpose-and-scope)

1.2 Review Process [1](#review-process)

1.3 Review Criteria [2](#review-criteria)

2.0 Program Summary [3](#program-summary)

2.1 Criterion 1: Approach to Self-Assessment
[3](#criterion-1-approach-to-self-assessment)

2.2 Criterion 2: Deployment of Self-Assessment
[8](#criterion-2-deployment-of-self-assessment)

2.3 Criterion 3: Results from Self-Assessment
[11](#criterion-3-results-from-self-assessment)

3.0 Conclusions 14

3.1 Strengths [14](#strengths)

3.2 Areas for Improvement [15](#areas-for-improvement)

3.3 IAP Management System Programmatic Issues
[15](#iap-management-system-programmatic-issues)

**Exhibits**

Exhibit 1 - Documents Reviewed [17](#__RefHeading___Toc68064789)

Exhibit 2 -- Personnel Interviewed 18

Exhibit 3 - Documents Referenced 19

## Introduction

### Purpose and Scope

This report presents the results of Independent Oversight's review of
the Self-Assessment (SA) Program of the Life Sciences Directorate. As an
element of the Integrated Assessment Program (IAP), Independent
Oversight (IO) is chartered to independently verify the effectiveness of
Brookhaven National Laboratory's Self-Assessment Program. To accomplish
this, IO personnel perform reviews of Directorates or other
organizations reporting to the Laboratory Director in accordance with
the requirements and guidelines provided in the IAP Management System
Description. Reviews are identified and scheduled in the FY 2004 Program
Plan: Review of Organizational Self-Assessment Programs \[FY04 IO Review
Plan\].

FY04 reviews focus on the organization's "approach" to self-assessment,
the status of "deployment" and the "use of results" to improve
performance. Specifically, IO will review how the organization is
planning to implement the self-assessment program, how the organization
conducts the self-assessment program, and how the results from
self-assessment are analyzed and used to improve performance. In the
course of this review, comparison of the current status of the Life
Sciences SA Program to that reported in **Independent Oversight Report
SA 01-05, Evaluation of the Self-Assessment Program of the Life Sciences
Directorate \[IO Report SA 01-05\]** was performed to measure progress
since the FY 2001 evaluation.

### Review Process

This review formally began on February 17, 2004, with an inbriefing
conducted with personnel from Life Sciences. This meeting served to
introduce the IO Reviewer, John Usher, who provided information on
review objectives, methodology, review criteria, and schedule. The
inbriefing also served to inform the IO Reviewer about Life Sciences and
to identify key personnel involved in the planning and implementation of
self-assessment.

The following data and collection methods were used during the review:

-   Documents Reviewed (Exhibit 1)

```{=html}
<!-- -->
```
-   Interviews (Exhibit 2)

```{=html}
<!-- -->
```
-   Documents Referenced (Exhibit 3)

Interviews were conducted using prepared questions tailored to address
the review criteria (see Section 1.3) in terms of key business
factors/performance objectives. Notes were prepared during each
interview to record the information gathered. Comments, concerns, and
follow‑up actions were also documented after each interview.

An outbriefing was conducted on April 8, 2004, with personnel from Life
Sciences. At this meeting, the IO Reviewer summarized results of the
review and submitted a draft report for review by Life Sciences
personnel. The meeting also gave Life Sciences personnel the opportunity
to provide feedback to the IO Reviewer on the review process.

### Review Criteria 

The IO assessment process evaluates an organization's self-assessment
program against the criteria published in the **FY04 IO Review Plan**.
These criteria are:

1.  Approach

```{=html}
<!-- -->
```
1.  The scopes of planned assessment activities are comprehensive, have
    a strong technical basis, and are balanced with work activities.

2.  The methods for conducting key scheduled assessments are defined and
    are commensurate with types of assessments planned and performance
    information desired.

3.  Management and staff involvement commensurate with their
    responsibilities is evident.

4.  Key supporting organizational processes (tracking/trending systems,
    causal analysis, critiques, etc.) and tools are developed.

5.  Organizational plans/procedures address regulatory and/or SBMS
    drivers for assessment activities.

```{=html}
<!-- -->
```
2.  Deployment

 

1.  Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.

2.  Assessments are documented and communicated as planned.

3.  Assessment results are evaluated/analyzed to a degree commensurate
    with the type of assessment. Strengths are identified.
    Corrective/improvement actions are identified, prioritized, assigned
    to specific owners, and assigned due dates. All conditions are
    tracked to closure. Actions are validated as improving performance.

4.  Evidence of timely self-identification of issues exists. Significant
    issues are brought to the attention of management and disclosed to
    regulatory/oversight agencies in accordance with contractual
    obligations.

5.  Management involvement is evident.

```{=html}
<!-- -->
```
3.  Results

```{=html}
<!-- -->
```
1.  Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations, such as mission
    achievement and retention/expansion of core competencies.

2.  Evidence exists that there is an appropriate connection between
    results of organizational self-evaluation and development of
    strategic/institutional plans.

Further detailed information supplementing the review criteria is
provided in the **FY04 IO Review Plan**
(<http://www.io.bnl.gov/safy04.doc>).

## Program Summary

This review covers the Life Sciences Directorate comprising the Biology
(BO) Department and the Medical (MO) Department.
Organizational/functional areas addressed in this review included the
Clinical Research Center (CRC), the NASA Space Radiation Laboratory
(NSRL), Radionuclide and Radiopharmaceutical (R&R) Isotope Production,
and Research Support (operations, ES&H, and business management). The
Interim Associate Laboratory Director (ALD) for Life Sciences does not
have stewardship responsibilities for any SBMS Management Systems.

A discussion and analysis of data collected for the review of the Life
Sciences Directorate is presented for each criterion. Areas that
demonstrated positive performance or programmatic strengths (criteria
exceeded) are identified with a (+). Areas where criteria are met but
not exceeded are unmarked. Areas that indicated opportunities for
improvement (criteria not met or partially met) are noted using a (-).

### Criterion 1: Approach to Self-Assessment

The Life Sciences Self-Assessment Manager (SAM) prepared the Life
Sciences FY04 Self-Assessment Plan \[SA Plan\]. The ALD and Department
Chairs approved the SA Plan. The SA Plan, Rev.0, is dated December 14,
2003.

The scope of the SA program is largely comprehensive in addressing the
Directorate's research, operations, and support services. Financial
management is not explicitly addressed in the SA Plan except for Isotope
Production. Financial management activities such as budget planning and
ensuring adherence to spending plans are evident. Appropriate
methodologies for assessment are described in the SA Plan. Assessment
activities are appropriately prioritized. Managers, staff, and key
stakeholders are appropriately involved in planning assessment
activities. Processes for supporting the organizational SA programs are
in place, although Directorate personnel do not use the Family
Assessment Tracking System (ATS) available to them. Evidence exists that
organizational managers appropriately consider SBMS, contractual, and
regulatory drivers in developing performance goals/objectives and in
planning and scheduling assessment activities, although not all FY04 BNL
required assessments relevant to Life Sciences were acknowledged in the
SA Plan.

1.  **The scopes of planned assessment activities are comprehensive,
    have a strong technical basis, and are balanced with work
    activities.**

Assessment activities are prioritized based on alignment with BNL and
organizational strategic plans, results of previous assessments,
business and operational risk, and available resources. In the SA Plan,
Section 1.0, Introduction and Purpose, the Plan is described as "a
derivative document based on guidance given in the Laboratory\'s
Integrated Assessment subject area, the Critical Outcome document, and
the goals listed in the Biology and Medical Departments' Mission
Statements." Key processes include "Directorate operational planning
based on the Laboratory\'s strategic goals and critical outcomes" and
"monitoring performance against these and other (Directorate-specific)
plans, objectives and performance measures." Attachment 1, FY04 Self
Assessment Activities, documents linkage of each objective to Critical
Outcome 1.0 or 3.0. Attachment 1 (p. 2) states, "Use data from Strategic
Planning Committees to develop, maintain and improve the quality and
quantity of research ..." and further states, "See Institutional Plan
(FY2004-2008) for further details." Both the Biology and Medical
Departments have instituted Planning Committees. These committees
function to provide advice and guidance to the respective Chairs.
Section 1.0 also states, "Future assessment planning and concomitant
activities will be based on previous assessment results/evaluations,
changes in the Directorate objectives, evolving Laboratory Management
System Initiatives, and current forecasts about the business climate."
Section 5.0, Implementation, states, "The year-end summary ... will be
used to set objectives for the next self-assessment cycle." While no
separate year-end summary was prepared, Life Sciences' contribution to
the FY03 BNL Year End Self-Evaluation documents the connection to FY04
objectives for science and technology. Section 5.0 also states, "In
addition to the plan outlined in \[Attachment 1\], ad hoc assessments
may be set up on an as-needed basis." There is no explicit mention of
risk in the SA Plan as regards identifying assessment topics. Section
5.0 states, "Existing operating budgets will be used to carry out the
tasks identified in the Self Assessment activities unless otherwise
specified."

**(+)** Life Sciences' SA Plan design is based on the IAP (Baldrige)
framework ensuring a comprehensive perspective of organizational
performance. Section 1.0 states, "... the overall assessment planning
and the evaluation criteria that Life Sciences ... utilized are based on
a framework built around the Malcolm Baldrige Award Criteria."
Attachment 1 specifically documents the connection to the IAP (Baldrige)
Assessment Planning and Evaluation Criteria Framework. Individual
objectives are explicitly linked to Criteria 1.0, 4.0, 5.0, and 6.0.

The SA Plan is largely comprehensive in that it covers the Directorate's
research, operational, business, and financial activities. The SA Plan,
Section 1.0, states "an integrated assessment program is designed to
generate information that can be used to make fact-based decisions on
improving the programmatic, operational and business performance within
the Laboratory." Research activities are detailed in Section 2.0,
Biology and Medical Department Mission Statements. Detailed performance
measures are documented under the "Objective: Maintain and improve the
quality and quantity of research conducted." Operational performance is
addressed under objectives such as: maintain and improve infrastructure,
ensure management oversight in operations, operate in compliance with
applicable laws and BNL regulations, conduct operations in accordance
with the expectations of BNL's Management Systems, conduct operations in
compliance with DOE's Radiation Control Program, effectively and
efficiently manage Clinical Research Center operations, assess Isotope
Production Program, and several environmental program objectives.
**(-)** Some business/financial activities are included in objectives
(e.g., Isotope Production) but there is no specific linkage to IAP
Framework Criterion 5.0 regarding financial performance, although it is
evident that managers and Principal Investigators interact with business
management personnel in reviewing budget and performance versus spending
plans.

The SA Plan evidences appropriate attention to customer
requirements/expectations. Section 1.0 states, "The Integrated
Assessment Program is structured to ... verify stakeholder and customer
expectations and requirements to improve science and technology
research, existing products and services and enhance customer
satisfaction." Attention to customer expectations is implicit in the
research performance objectives and measures. The expectations are
explicitly stated in DOE Field Work Proposals (FWPs) and
proposals/grants documenting Work for Others (WFOs) agreements. Section
5.0, Implementation, states, "The Directorate has always undergone
self-assessment through its various internal committees that address
specific areas (ES&H, Support Staff Review, Continuing Appointments
Review, etc.) as well as external committees (advisory, visiting, DOE
Review, etc)." These internal and external committees represent an
opportunity for input and feedback from customers. For example, Life
Sciences Directorate personnel participated in a Strategic Review of
Biology and Medical Department Programs \[Strategic Program Review\] on
February 11 & 12, 2004. This Review was conducted at the behest of
program managers from DOE's Office of Biological and Environmental
Research (BER) originating in a DOE BER Program Review conducted in
November 2003. Attachment 1 contains no explicit linkage to IAP
Framework Criterion 3.0, Customer Focus and Satisfaction (Customer
Value); although one objective addresses "the Laboratory's relationship
with stakeholders." It is noted that customer comments/complaints
regarding the Isotope Production Program are tracked and addressed by
R&R personnel. A single complaint was addressed as documented in the
FY03 BNL Self-Evaluation. The CRC administers a survey for human
subjects upon the completion of protocols. Results from these surveys
are reported to the Quality Assurance, Care & Safety Committee (QAC&SC).

While the SA Plan, Section 1.0 states, "Future assessment planning and
concomitant activities will be based on previous assessment
results/evaluations," there is no explicit mention of considering the
effectiveness of previously implemented actions during planning. Section
3.0, Roles and Responsibilities, states that the SAM tracks "the
implementation of appropriate corrective actions." Section 6.0,
Corrective, Preventative, and Improvement Action Management,
acknowledges that the Directorate follows BNL ESH Standard 1.2.1, which
states, "Managers should consider the potential significance of the
corrected condition(s) and decide on the need for a review (at a future
date) of the effectiveness of completed action(s)."

Results from external assessments are incorporated into the
Directorate's self-assessment and annual self-evaluation process. As
discussed previously, the SA Plan, Section 5.0, incorporates assessment
by "external committees (advisory, visiting, DOE Review, etc)."
Attachment 1 specifically includes the DOE Program Review of Life
Sciences; topical or management system assessments led by other BNL
organizations, DOE, and/or regulators; ISO 14001 Registration Audit; and
this IO Review. It is evident that the CRC has appropriately considered
requirements from accreditation by the Joint Commission on Accreditation
of Healthcare Organizations (JCAHO) including, explicitly, the
requirement to conduct Periodic Performance Reviews. The Life Sciences
Directorate also responded appropriately to the IO FY01 evaluation of
the Life Sciences self-assessment program and IO independent assessments
covering benchtop experimental process and the use of controlled
substances in research, for example. During Tier I inspections, it was
confirmed that Life Sciences has responded and continues to address
items noted during the recent OSHA inspection of BNL.

External lessons learned are incorporated into the organization's
self-assessment process. Life Sciences personnel subscribe to the BNL
Lessons Learned service. **(+)** The Directorate makes effective use of
external peer/expert reviews to incorporate external experience into
scientific, operational and safety-related aspects of the Directorate.
Additionally, personnel also participate on external committees, are
active members of associations/collaborations, and attend conferences
and workshops.

2.  **The methods for conducting key scheduled assessments are defined
    and are commensurate with types of assessments planned and
    performance information desired.**

Assessment activities are assigned to qualified personnel including
appropriate use of subject matter experts. The SA Plan, Attachment 1,
states, "Maintain Tier I program including the regular participation of
ES&H subject matter experts \[SMEs\]." In periodically assessing CRC
operations, "utilize independent subject matter experts in the conduct
of selected assessments." Environmental Compliance Representatives
(ECRs) are involved in many aspects of environmental management. The IO
Reviewer observed the participation of SMEs on two Tier I inspections.
**(+)** Tier I program uses as performance indicators the number of SMEs
invited and the number of disciplines represented.

Managers effectively communicate their expectations such that all
employees understand their roles and responsibilities for
self-assessment. The SA Plan, Section 3.0, clearly documents roles and
responsibilities regarding the self-assessment program for the ALD,
Department Chairs, line managers, staff, and the SAM. Section 5.0
states, "Input for the contents of the \[SA Plan derives from\] the
Biology and Medical Department Chairmen, the SAM and other members of
the staff (scientific, technical, administrative-safety,
administrative-business)." Attachment 1 is "used to drive the
self-assessment process and set responsibilities, with individuals or
committees to evaluate performance or collect information." Attachment 1
identifies "the person, position or group responsible for specific
self-assessment measures. They will conduct the appropriate tasks and
report the status of those tasks to the SAM. They will work with the SAM
to analyze the data collected for inclusion in a year-end Directorate
summary. The responsible person is also responsible for tracking and
assuring the implementation of any corrective, preventative or
improvement actions."

Managers ensure that independent assessments and peer reviews are
considered and incorporated into the self-assessment process, as
appropriate. Independent assessments conducted by management system
stewards (Environmental Management System \[EMS\], Integrated Assessment
Program, Work Planning and Control) are explicitly included in the SA
Plan. The Life Sciences Directorate makes extensive use of peer and
management reviews of proposals and documents submitted for publication.
The SA Plan also acknowledges assessment by "external committees
(advisory, visiting, DOE Review, etc)" serving as peer review. The
Strategic Program Review, mentioned previously, included peers from
universities as well as other DOE national laboratories. During FY2003,
A peer group of National Institutes of Health funded researchers
reviewed the operations (by BO) of beam lines at the National
Synchrotron Light Source; DOE accepted the results of this peer review
instead of performing their own review.

The assessment approach uses a mix of information sources including
performance observation. Approaches (methodologies) listed in Attachment
1 include research program reviews (preparation for presentations
including sponsor/peer review), conducting operational/exit readiness
reviews, Tier I inspections, management walkthroughs, EMS assessments,
EMS management review, periodic management/staff meetings, environmental
compliance assessments, reviews of process evaluations, experimental
reviews, work planning, environmental aspects analysis, topical
assessments (controlled documents, Title V, SPDES, beryllium,
peroxidizables, Chemical Management System, work planning --
verification of compliance with requirements), review of Job Training
Assessments, assessment of lab animal training program, assess maturity
of self-assessment program, review environmental monitoring data,
performance indicators, root cause analysis, review inventory of
regulated etiologic agents, assess implementation of CRC protocols,
assess CRC program, assess status of controlled substance program,
progress reports, cost control, customer feedback, and Current Good
Manufacturing Practice (cGMP) assessments.

The assessment approach minimizes the impact to work activities.
Assessments are planned/scheduled in advance and communicated to
affected personnel. Tier I intends to involve affected personnel since
it is desirable to interact with personnel in their work spaces.

3.  **Management and staff involvement commensurate with their
    responsibilities is evident.**

Planning involves managers and staff members as appropriate to ensure
that all aspects of the organization's operations are evaluated. The SA
Plan, Section 3.0, states, "\[The\] SAM ... is responsible for drafting
the self-assessment program with members of the Department and
submitting it to the Department Chairs and then the ALD, Life Sciences
for final approval." Managers also participate through "Directorate
operational planning." Managers and staff have specific responsibilities
per Section 3.0. Section 5.0 states, "Input for the contents of the \[SA
Plan derive from\] the Biology and Medical Department Chairmen, the SAM
and other members of the staff (scientific, technical,
administrative-safety, administrative-business)." Specific SA Plan
attachments were contributed by R&R personnel and the CRC Manager.

The SA Plan, Rev. 0, is dated 12/14/03, thus approved prior to 12/31/03
according to BNL's required schedule. The SA Plan was reviewed by
managers, the ESH Coordinator, and the Environmental Compliance
Representative (ECR). The SA Plan was approved by Department Chairs and
the ALD.

4.  **Key supporting organizational processes (tracking/trending
    systems, causal analysis, critiques, etc.) and tools are
    developed.**

The organization has a defined process for analyzing the results of
assessment activities. Section 3.0 states, "The SAM will analyze this
data against the Directorate's goals and objectives and track the
implementation of appropriate corrective actions based upon the
analysis." Section 5.0, under Responsibility, states, "The person,
position or group responsible for specific self-assessment measures ...
will work with the SAM to analyze the data collected for inclusion in a
year-end Directorate summary." Tier I process tracks the number of
"repeat findings each quarter." Managers are attuned to the term
"repeat" and respond accordingly. **(+)** Attachment 1, Objective:
Conduct hazardous waste generating operations in compliance with RCRA,
states, "Conduct root cause analysis on hazardous waste generation in
benchtop research areas." Tier I inspections and other assessment
activities have identified issues with management of waste in Satellite
Accumulation Areas, as also noted prior to a recent Resource
Conservation and Recovery Act (RCRA) inspection. Causal analysis will be
conducted in concert with other "small science" organizations. The
Isotope Production Program maintains "a tracking system for customer
feedback." Customer comments/complaints are tracked and analyzed.

The organization has multiple processes for tracking corrective,
preventive, and improvement actions to completion. Section 3.0 states,
"The SAM will gather status and data information from those responsible
for self-assessment activities. The SAM will analyze this data against
the Directorate's goals and objectives and track the implementation of
appropriate corrective actions based upon the analysis." The SAM
maintains a spreadsheet of actions. Section 5.0, referring to "the
person, position or group responsible for specific self-assessment
measures" as listed in Attachment 1, states, "The responsible person is
also responsible for tracking and assuring the implementation of any
corrective, preventative or improvement actions." As part of the Tier I
process, action status is reviewed during the next periodic inspection.
The quality representative for the R&R Group tracks action status using
a tabular format. Status of Operational Readiness Evaluation findings is
also tabulated. Also, referring to the "person, position or group
responsible for specific self-assessment measures," Section 5.0 states,
"\[Life Sciences\] tracks this type of routine activity in the Family
ATS system or other tracking system." To date, the Life Sciences
Directorate has not used the Family ATS but does have a tracking system
in place. Section 6.0 states, "The Directorate follows ESH Standard
1.2.1, Corrective Action Management and Tracking for Internal and
External Assessments." Attachment 1, Objective: Operate in compliance
with applicable laws and BNL regulations, under Environmental
Compliance, states, "Identify and close out remaining Corrective Actions
identified in the process evaluations and track to completion in the
Environmental Services Division (ESD) Family ATS." The same objective
contains as a performance indicator "% Corrective Actions Completed."
The IO Reviewer noted that actions were tracked in the ESD Family ATS
and in the tracking system used by the SAM.

Records are maintained that demonstrate effective planning, timely
completion of assessments, and analysis of results. Evidence of this
includes the SA Plan, Life Sciences' contributions to the BNL Annual
Self-Evaluation Report, and specific assessment reports, for example,
Tier I Safety Inspection memoranda, The Worker Safety and Health
Required Assessment Aid (FY03), Routine cGMP Assessment memoranda,
QAC&SC minutes, CRC Protocol Compliance Monitoring Records, and
Summaries of \[Human\] Subject Surveys and Follow-up.

Some available information systems are used to support assessment
activities. The Integrated Information Management System (IIMS) provides
a repository for Critical Outcome 1.0 information. Life Sciences
websites contain internal procedures. Institutional ATS (external and/or
independent assessments per ESH 1.2.1) and ESD Family ATS are used to
track some Life Sciences assessment activities as discussed previously.

5.  **Organizational plans/procedures address regulatory and/or SBMS
    drivers for assessment activities.**

Assessment activities are based on and linked to the organization's
performance objectives and measures as well as the BNL Critical Outcomes
and associated Performance Objectives/Measures. The SA Plan, Attachment
1, matrixes assessment activities to specific organizational objectives.
Attachment 1 also documents linkage of specific organizational
objectives to BNL Critical Outcomes.

"Required Assessments" are identified in the SA Plan demonstrating
appropriate consideration of integration of SBMS Management System
requirements. Section 1.0 states that the assessment program is
structured to "validate conformance to established internal and external
requirements." Attachment 1 includes an indication of whether specific
assessment activities (measures) are required or not. Attachment 1,
Objective: Conduct operations in accordance with the Expectations of the
BNL Management Systems." The list of required assessments includes a
comprehensive perspective for ES&H required assessments. Tier I,
Operational (and Exit) Readiness Evaluations, and Environmental
Assessments are identified in the SA Plan. **(-)** The SA Plan, Rev. 0,
does not include some required assessments posted in the Integrated
Assessment subject area in January 2004. Not included are required FY04
self-assessments addressing Fire Safety, Maintenance Management, Energy
Management Awareness, Level 2 Managers' Self-Assessment of Records
Management System, Foreign Visits and Assignments, and LOTO. The SAM had
added these required assessments to a draft revision of the SA Plan and
accurately stated that assessment aids for these required assessments
were not available at the time the SA Plan was initially issued. The IO
reviewer verified that selected FY03 required assessment aids (Building
and Facility Management, Life Cycle Asset Management, Safeguards and
Security, and Worker Safety and Health) were completed by Life Sciences.
The Work Planning and Control Management System point of contact also
conducted assessments of Satellite Pharmacy Inventories and
Authorization for Use of Controlled Substances in Research.

### Criterion 2: Deployment of Self-Assessment

Assessment activities are conducted as scheduled. Schedules may be
adjusted based on changing priorities as determined by organizational
managers. Results of assessment activities are documented and
communicated to responsible Life Sciences managers for review and
analysis. Key assessment results are regularly discussed and analyzed at
QAC&SC and/or management meetings. Managers evaluate assessment results
to identify both strengths and opportunities for improvement. Necessary
improvement actions are identified and communicated to responsible
managers and stakeholders.

1.  **Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.**

Assessment activities for FY03 and those scheduled so far in FY04 have
been completed as scheduled. Performance information related to Critical
Outcome 1.0 was collected and reported to IIMS as scheduled. The
Strategic Program Review was conducted as scheduled in February 2004.
FWPs were recently prepared, reviewed, and submitted per schedule. The
FY03 EMS and Operations Management Review was completed in September
2003. The IO Reviewer observed two quarterly Tier I inspections that
were conducted as scheduled. Life Sciences participated in BNL's
internal EMS assessment as scheduled in February 2004. The cGMP annual
assessment was conducted ahead of schedule. Several other activities
(e.g., CRC assessments, material inventories) are ongoing as planned.

The SA Plan, which is managed as a controlled document, will be updated
to reflect changes in scope. The SA Plan is Rev. 0 to date, but the SAM
has indicated intent to update the SA Plan to incorporate FY04 required
assessments attached to the Integrated Assessment subject area, for
example.

Customer feedback is solicited as part of the organization's assessment
activities. The SA Plan, Section 5.0, states, "The Directorate has
always undergone self-assessment through its various internal committees
that address specific areas (ES&H, Support Staff Review, Continuing
Appointments Review, etc.) as well as external committees (advisory,
visiting, DOE Review, etc)." These internal and external committees
represent an opportunity for obtaining feedback from customers.
Attachment 1 contains no explicit linkage to IAP Framework Criterion
3.0, Customer Focus and Satisfaction (Customer Value); although one
objective addresses "the Laboratory's relationship with stakeholders."
As discussed previously, customer comments/complaints are tracked for
the Isotope Production Program. The CRC surveys human subjects following
completion of protocols.

2.  **Assessments are documented and communicated as planned.**

Assessment activities are documented and results are provided to
responsible managers. Section 3.0 states, "The SAM will gather status
and data information from those responsible for self-assessment
activities. The SAM will analyze this data against the Directorate's
goals and objectives and track the implementation of appropriate
corrective actions based upon the analysis. The SAM will provide a
year-end summary to the Department Chairs and the ALD for Life
Sciences." The SAM documented the results of the FY03 EMS and Operations
Management Review and distributed this to Life Sciences management.
While no separate year-end summary was prepared for science and
technology performance, the SAM entered performance information for
Critical Outcome 1.0 into the IIMS where it was reviewed and approved by
Life Sciences management. Section 5.0, under Responsibility, states,
"The person, position, or group responsible for specific self-assessment
measures ... will conduct the appropriate tasks and report the status of
those tasks to the SAM." In many cases, the responsible person is the
line manager. The IO Reviewer noted completed reports for Tier I
inspections, Isotope Production assessments, and CRC assessment
activities that were appropriately distributed to management.

The assessment schedule is managed to ensure information is available to
support timely management decisions. Assessment activities addressing
research program performance are timed to permit submittal to IIMS per
BNL's schedule for submission of the annual Science and Technology Year
End Self-Evaluation \[S&T Report\] to DOE. The FY03 S&T Report is dated
October 8, 2003. The assessment schedule as documented in the SA Plan is
timed to conclude other activities before the end of the fiscal year.
FWPs were submitted in February to allow for timely preparation of
budgets for FY05.

3.  **Assessment results (internal and external) are evaluated/analyzed
    to a degree commensurate with the type of assessment. Strengths are
    identified. Corrective/improvement actions are identified,
    prioritized, assigned to specific owners, and assigned due dates.
    All conditions are tracked to closure. Actions are validated as
    improving performance.**

The Life Sciences Directorate has an effective process for identifying
opportunities for improvement. The SA Plan, Section 1.0, states that the
assessment program is structured to "provide ... performance information
to promote early identification and resolution of problems" and to
"contribute to ongoing performance improvement." Section 1.0 also states
that key processes of the SA Plan include "evaluating overall
performance and identifying areas for improvement." Section 3.0 states
that Chairs are "responsible and accountable for ensuring that
appropriate corrective action to identified weaknesses is taken."

Improvements are prioritized in accordance with business and operational
risks and cost benefit. Section 3.0 states that Chairs are "responsible
and accountable for ensuring that appropriate corrective action to
identified weaknesses is taken, commensurate with hazard, risk,
vulnerability, and overall business objectives." Section 7.0,
Prioritization, states, "Priority is considered when establishing due
dates for action." **(+)** Attachment 1, Objective: Maintain and Improve
Infrastructure, includes a list of projects identified as "key areas"
and linked through strategic planning. The risk of noncompliance with
requirements is specifically addressed in Attachment 1, Objective:
Operate in compliance with applicable laws and BNL regulations. Risk of
nonconformance to BNL's Management Systems is specifically addressed in
the Objective: Conduct operations in accordance with the Expectations of
BNL's Management Systems.

Improvement actions are developed with consideration of causal factors
as commensurate with the level of hazard/risk. Section 4.0 states that
the self-assessment process entails "the analysis of \[performance\]
data against the Directorate's objectives, and ... the implementation of
appropriate corrective actions based upon the analysis." The SBMS
Subject Area: Corrective and Preventive Action describes the process of
causal analysis. **(+)** As a notable example, Attachment 1 (p. 13)
states, "The majority of Tier I inspections find deficiencies in the
management of Satellite Accumulation Areas for hazardous waste. In an
effort to reduce those, we will work with other benchtop research Depts.
(in EENS) to conduct a causal analysis on the management of Hazardous
Waste by the generators."

Change control for action due dates reflects consideration to balance
priorities. The SA Plan, Section 6.0, states, "The Directorate follows
ESH Standard 1.2.1, Corrective Action Management and Tracking for
Internal and External Assessments." ESH 1.2.1 states, "If actions cannot
be completed by the assigned due date, the owner (with Assessment owner
concurrence) can extend the due date with justification ..."

Improvements are validated, as commensurate with the level of
hazard/risk, to ensure desired outcomes are achieved including
prevention of recurrence of adverse events. Section 3.0 states that
Chairs are "responsible and accountable for ensuring that appropriate
corrective action to identified weaknesses is taken, commensurate with
hazard, risk, vulnerability, and overall business objectives." Section
5.0 states that the responsible person, as identified in Attachment 1,
"is also responsible for tracking and assuring the implementation of any
corrective, preventative or improvement actions." Tier I inspections
routinely determine whether or not previous findings have been
successfully addressed. Some Tier I findings are addressed at the time
of the inspection. The IO Reviewer noted that findings and actions
resulting from the OSHA inspection at BNL were discussed and reviewed
during Tier I inspections. Life Sciences tracks the percentage of
corrective actions "identified in any new process evaluations"
completed. CRC Protocol Compliance Monitoring actions are followed up by
the CRC Manager. Follow-up independent assessments have been used to
verify closure of corrective actions and/or to validate effectiveness of
actions taken. One example of this was IO's follow-up to the Work
Planning and Controls Management System assessment of use of controlled
substances in research.

4.  **Evidence of timely self-identification of issues exists.
    Significant issues are brought to the attention of management and
    disclosed to regulatory/oversight agencies in accordance with
    contractual obligations.**

Results are analyzed to identify potential noncompliances with nuclear
safety rules (Price-Anderson Amendments Act). There is no specific
mention of PAAA in the SA Plan; however, the Special Assistant to the
ALD for Life Sciences is a member of the PAAA Working Group who is
highly cognizant of PAAA issues. In Attachment 1, the Objective: Operate
in compliance with applicable laws and BNL regulations, lists Radiation
Control Program and Quality Management as two areas with assessment
activities. Attachment 1 also contains the Objective: Conduct operations
in compliance with DOE's Radiation Control Program. Facility Support
personnel write Radiological Awareness Reports, which are reviewed by
PAAA Coordinator. Occurrence Reports are also reviewed by the PAAA
Coordinator. Facility Support Representatives also conduct quarterly
self-assessments (as required by the Radiological Control Division's SA
Plan) in the Life Sciences Directorate.

Results are analyzed to determine potential impacts on SBMS documents
and internal policies and procedures. Section 5.0 acknowledges "routine
review and updates of organizational documents" including "R2A2s,
experimental review documents, environmental aspects analysis, and
process evaluations." Attachment 1 specifically discusses participating
in a topical assessment of the Controlled Document subject area,
participation in assessments of Work Planning and Control, revision of
the Using Controlled Substances in Research subject area, and ongoing
involvement with the Biosafety in Research subject area and the EMS.
Life Sciences personnel participate on teams developing SBMS Subject
Areas.

Results of assessment activities have been appropriately considered for
publication in the BNL Lessons Learned Program. Life Sciences personnel
have authored BNL Lessons Learned and gained recognition for their
efforts. As a specific example, Attachment 1 (p. 13) states, "The
majority of Tier I inspections find deficiencies in the management of
Satellite Accumulation Areas for hazardous waste. In an effort to reduce
those, we will work with other benchtop research Depts. (in EENS) to
conduct a causal analysis on the management of Hazardous Waste by the
generators. We will also communicate to the staff the importance of RCRA
compliance and the problems we find (lessons learned)."

5.  **Management involvement is evident.**

Managers participate in planning and conducting assessment activities.
The SA Plan, Section 3.0, states, "Line Managers are responsible for
conducting an assessment of research activities, facilities, and
operations under their purview." Line management participation in
planning for these assessment activities is implicit. Senior managers
from Life Sciences and BNL participated in planning and rehearsals for
presentations given at the recently concluded Strategic Program Review
held at BNL. These rehearsals provided senior managers with the
opportunity to review research performance and to develop plans for
future endeavors. The ALD and the BO Chair also provided summary
presentations at this Review demonstrating thorough knowledge of the
status of research and operations, human resources, and budgets.

The SA Plan, Attachment 1, lists responsible persons who will conduct
the assessment activities. The ALD, Chairs, and Principal Investigators
are listed as responsible for several assessment activities (measures)
including "senior manager walkthroughs." Senior manager walkthroughs are
not routinely documented. The ALD has observed work in progress (e.g.,
critical lift of the microMRI). The ALD also visited with individual
researchers in their laboratory spaces to discuss research
activities/plans. The BO Chair conducts almost daily walkthroughs of
different areas of Bldg. 463. The BO Chair also participates in
approximately one third of the Tier I inspections in Bldg. 463 or about
one per quarter. The Special Assistant to the ALD participated in one of
the Tier I inspections observed by the IO Reviewer. The R&R Manager
attended a recent BNL Senior Management Work Observation at the
Brookhaven LINAC Isotope Producer.

Life Sciences managers participate in the analysis and evaluation of
assessment results. Section 5.0 states the responsible person "will work
with the SAM to analyze the data collected for inclusion in a year-end
Directorate summary." The ALD/Chairs review and approve the information
that is entered into IIMS regarding Critical Outcome 1.0. Life Sciences
managers and the BNL Deputy Director for Science and Technology interact
to develop the final evaluation of performance.

The SA Plan, Section 3.0, states, "Chairmen are also responsible and
accountable for ensuring that appropriate corrective action to
identified weaknesses is taken ...." Section 5.0 states, "The
responsible person is also responsible for tracking and assuring the
implementation of any corrective, preventative or improvement actions."
Managers are involved to varying extent in validating the effectiveness
of actions through the process discussed in section 2.2.3 previously.

### Criterion 3: Results from Self-Assessment

The Life Sciences Directorate provided input to the FY03 BNL Year End
Self-Evaluation Report as in previous years summarizing performance
results for Critical Outcome 1.0 for both BO and MO. Critical Outcome
performance reporting was accomplished via computerized submittals to
IIMS. The Life Sciences Directorate also documented performance in the
EMS & Operations Management Review Minutes. Reports of performance also
described improvements achieved and established a basis for planning.
Results are communicated to BNL senior management. There is appropriate
linkage between the SA Plan and BNL's Institutional Plan.

1.  **Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations, such as mission
    achievement and retention/expansion of core competencies.**

Progress towards achievement of near-term and long-term goals and
objectives is evident. Objectives for improving quality and quantity of
research, improvement of infrastructure, and Isotope Production Program,
for example, are evidence that progress is sought. The BNL FY03 Year-end
Report documents (Section 1.3.3.1) achievement of the scientific and
technical objectives contained in Life Sciences' FY03 SA Plan. Summary
presentations at the recent Strategic Planning Review also highlighted
these and other achievements. Strategic hires of scientific personnel
are evident in both BO and MO. Notably, the overall health of BO
research programs has improved markedly since the last IO review in FY01
according to interviewees. Also, in pursuit of additional funding from
sponsors other than DOE, the Directorate has improved the grant/proposal
preparation process by designating a senior scientist as mentor to those
preparing proposals and by acquiring the services of a project
management expert to prepare FWPs. Directorate personnel have also
attended program development workshops. The Directorate reports
achieving increased success with percentage of proposals funded.

The BNL FY03 Year-end Report (Section 1.3.3.3) and Strategic Program
Review summary presentations also describe how conduct of
operations/research has been enhanced in the Life Sciences Directorate.
As an example, the NSRL became operational and a pilot experimental run
was conducted followed by the first full-scale experimental run; user
feedback was positive and some improvements have been implemented for
the second experimental run taking place currently. **(+)** NSRL
end-of-run Reports provide a summary of experiments and operations that
reflect performance. The Imaging Program has enhanced collaborative
efforts with the National Institutes of Health (intramural program at
BNL), Ames Laboratory, and Lawrence Berkeley National Laboratory. The
CRC is increasing collaboration with Stony Brook University (including
joint appointments) and is pursuing operation as a General Clinical
Research Center. Sustained excellence is evident as the Brookhaven
Laboratory Animal Facility received favorable reviews from the
Association for the Assessment and Accreditation of Laboratory Animal
Care International, the U.S. Department of Agriculture, The New York
Sate Department of Health, and internally by the Institutional Animal
Care and Use Committee. The BNL FY03 Year-end Report also states, "The
Clinical Research Center completed its corrective action plan from a
shutdown of operations in 2000. There were numerous improvements to
operations including a restructuring of the program. The clinical
program has undergone several audits in FY03 all of which have had
favorable results."

It is also evident that customers and other stakeholders value the
research, products, and services of the Life Sciences Directorate. The
CRC surveys human subjects after completion of protocols. Two recent
reports of survey results indicate less than 4% of subjects had
complaints regarding service. The CRC also follows up with "protocol
study subjects within 24 hours of their research study." CRC reports
that only a very small percentage of subjects had complaints.
Additionally, only one complaint was received regarding Isotope
Production and this was addressed. Life Sciences' managers have received
some favorable feedback from the Strategic Program Review conducted in
February 2004, but the detailed report of the Review Committee has not
yet been received. Life Sciences' managers expressed a keen interest in
receiving this report as soon as possible. \[Note: A draft report was
received by Life Sciences after interviews were completed.\]

In addition to examples discussed previously, the Life Sciences
Directorate also provided examples of other improvements resulting from
the self-assessment process including improvements to the
self-assessment process itself. Section 1.0 states, "\[The SAP\] also
includes the Directorate's environmental objectives and the respective
action plans necessary to achieve these objectives. These additions
reflect one of the Directorate's improvement initiatives to streamline
the EMS Program since obtaining ISO 14001 registration. Execution of
this self-assessment plan provides the Directorate one of several
mechanisms for ensuring that Directorate standards and procedures are
consistent with applicable requirements and provisions of the Lab-level
ESH&Q program."

2.  **Evidence exists that there is an appropriate connection between
    results of organizational self-evaluation and development of
    strategic/institutional plans.**

Directorate/Department plans are modified as necessary based on
evaluation of assessment results and evidence consideration of DOE
Strategic Plans and linkage to the BNL Institutional Plan. The SA Plan,
Attachment 1, states, "Use data from \[Departmental\] Strategic Planning
Committees to develop, maintain and improve the quality and quantity of
research ..." and further states, "See Institutional Plan (FY2004-2008)
for further details ...." Attachment 1, regarding the Isotope Production
Program, states, "Strategic planning: production site manager
participates in semi-annual strategic planning sessions for program,
assistance in the preparation of a 5 yr. business plan which includes a
yr. 1 budget, quarterly progress reports and an annual report are
submitted which includes strategic planning as well as budget and
productivity reporting (against performance measures)."

Improvement actions are either acted upon at the
organization/Directorate level or communicated to BNL senior management
to be included in the Laboratory's improvement agenda, as appropriate.
The SA Plan, Section 1.0, states that the assessment program is
structured to "contribute to ongoing performance improvement, including
driving the Laboratory improvement agenda through the identification of
strategic and tactical investments and changes to the Critical
Outcomes." Section 5.0 states, "Life Sciences' \[science and
technology\] Performance measures which support Critical Outcome \[1.0\]
are included in and tracked through the Laboratory's Integrated
Information Management System (IIMS)." In providing semi-annual reports
to IIMS, \[Life Sciences'\] management communicates status of
achievement of performance objectives and identifies opportunities for
improvement to BNL senior management. Life Sciences' managers also meet
with the Deputy Director for Science and Technology to jointly evaluate
Directorate performance to facilitate the determination of scores for
the elements of Critical Outcome 1.0. The Life Sciences Directorate's
input to the BNL Institutional Plan also accomplishes this
communication. BNL senior management was also involved in the
preparation for the recently completed Strategic Program Review
including the rehearsals for presentations to the Review committee.
Additionally, The Assistant Laboratory Director for Environment, Safety,
Health & Quality and the Manager of the Environmental and Waste
Management Services Division attended the Life Sciences EMS & Operations
Management Review.

In addition to the annual BNL self-evaluation process, Life Sciences
demonstrated responsiveness to other independent or external assessments
as exemplified via improvements addressing the Area for Improvement and
the Recommendation included in **IO Report SA 01-05**:

-   **The Self-Assessment Plan does not include Mission Statement and/or
    "scientific" performance goals for the Medical Department.** The
    FY04 SA Plan incorporates a detailed description of the Medical
    Department's mission. Section 2.2 states the mission of the
    Department and includes "major, long term research goals" in
    Radiation Treatment, Imaging, Radioisotopes, Instrumentation, and
    Radiobiology.

Life Sciences also responded appropriately to BNL Independent Oversight
Report IO 03-12 by implementing recommended improvements related to the
process for use of controlled substances in research at BNL (ATS 1463).

## Conclusions

### Strengths 

The Life Sciences Directorate evidences improvement in the
organizational Self-Assessment Program since FY01, for those elements
reviewed previously, over what was already recognized as an effective
program. The SA Program is formally documented and comprehensive in
scope. Approaches to self-assessment activities include more than just
traditional audits and represent a performance-based concept. Roles and
responsibilities for self-assessment are clearly defined. Processes for
documenting results of activities, analyzing results, identifying
opportunities for improvement, and tracking actions are in place. The SA
Plan documents an annual cycle of continuous improvement in
effectiveness and efficiency, represents a forward-looking approach, and
provides linkage with the BNL Institutional Plan in planning and
achievement of longer-term goals and objectives.

-   Life Sciences Directorate managers interviewed uniformly articulated
    commitment to improving organizational and individual performance as
    reflected in the research, services, and/or products provided by the
    Directorate. As mentioned previously, specific improvements are
    documented in the FY03 BNL Year End Self-evaluation, presentations
    for the Strategic Program Review, and the EMS & Operations Minutes.
    Notably, managers indicated a desire for further improvement even in
    areas that were highly rated by customers/stakeholders.

```{=html}
<!-- -->
```
-   The Life Sciences Directorate makes effective use of different types
    of peer review both internal to the Directorate and external to the
    Directorate and BNL. The extensive preparations that preceded the
    Strategic Program Review demonstrated an effective peer review
    process that led to an effective set of presentations by both
    Departments. The Strategic Program Review was conducted by peers
    from DOE National Laboratories and universities. DOE Program
    managers also attended and observed the Review. Life Sciences
    personnel participate in numerous BNL committees and external
    conferences, workshops, and/or associations as another means of
    acquiring information from peers.

-   The format of the SA Plan, particularly as reflected in Attachment
    1, provides an exemplary model that goes beyond the requirements of
    the Integrated Assessment subject area. The Attachment shows linkage
    to Critical Outcomes and the IAP framework. The matrix format of
    Attachment 1 clearly enables interaction between the SAM and
    responsible managers and specifically ensures the identification of
    objectives for each of the diverse areas covered in the
    comprehensive SA Plan. The individual matrices effectively offer
    one-stop shopping for managers and stakeholders interested in
    science and technology, infrastructure, ES&H, Isotope Production,
    the CRC, etc.

-   Infrastructure objectives are specifically documented in the SA
    Plan. Attachment 1 documents the intent "to identify key areas
    (linked through strategic planning) for infrastructure improvements
    and apply for \[General Plant Projects\] funding or other applicable
    funding." Previous success in acquiring funding for infrastructure
    projects is noted. During a previous assessment, IO acknowledged the
    effective involvement of the BO Chair in the pursuit of funds for
    such projects.

-   NSRL end-of-run reports provide an effective means of describing
    results from each experimental run period and also a basis for
    evaluating the effectiveness of improvements made from one run to
    the next.

-   The combined EMS and Operations Management Review represents an
    effective and efficient means of reporting and evaluating
    performance as well as communicating these results to Life Sciences
    and BNL management.

-   Life Sciences has articulated a desire to increase the participation
    of subject matter experts (SMEs) on Tier I inspections. The
    Directorate rotates areas of focus for Tier I inspections to make
    effective use of the particular SMEs participating. Notably, the
    Directorate also tracks SME participation (% of Tier Is) as a
    performance indicator.

-   The recognized need to analyze performance of satellite waste area
    managers led to a proactive approach to resolving identified issues.
    The intent to work in concert with other "small science"
    organizations represents an efficient approach for the Directorate
    and for BNL.

### Areas for Improvement 

-   While it is clear that assessment activities covering aspects of
    business and financial management in Life Sciences are occurring,
    these activities (and corresponding objectives) are not documented
    in the SA Plan.

> **Recommendation:** The SAM indicated intent to incorporate budget and
> spending activities in Attachment 1. The SAM should ensure that
> business/financial management is incorporated to fully address IAP
> Framework Criteria 5.1 "Performance against Key Objectives and
> Measures (Financial, Operational, Programmatic)."

-   The SA Plan, Rev. 0, does not include all required assessments for
    FY04. The SAM correctly noted that the Integrated Assessment subject
    area was not revised to attach FY04 required assessments until after
    December 31 when SA Plan updates were due.

> **Recommendation:** The SAM should review required assessments and
> determine which apply to Life Sciences and incorporate these
> assessments in the next revision of the SA Plan.

-   The SA Plan does not specifically include some other assessment
    activities (JCAHO accreditation, AAALC accreditation, NSRL
    end-of-run, etc.) that are conducted in Life Sciences.

> **Recommendation:** The SAM should ensure that a comprehensive list of
> assessment activities is included in the SA Plan. Life Sciences
> managers can benefit from this list in ensuring the adequate
> distribution of resources to both conduct and respond to assessment
> activities. A multi-year schedule may be of value to acknowledge that
> certain activities (e.g., accreditations) do not occur every year but
> are part of overall performance measurement.

### IAP Management System Programmatic Issues

It is acknowledged that some of the issues discussed below are known to
BNL management and the IAP Management System Steward and Point of
Contact, and that efforts may be underway to address these issues. These
issues are noted here for emphasis.

An interviewee expressed concern about the timing (January 2004) of the
most recent revision to the Integrated Assessment Subject Area
incorporating new required assessments and the need for organizations to
update their SA Programs/Plans shortly after initial issuance/approval
as required by December 31 each year. This flawed timing creates the
appearance of organizations' being out of conformance to the Integrated
Assessment subject area while awaiting the announcement of line
organization assessments required by Management System Stewards.

At the time of this IO Review, Life Sciences had not received written
feedback from the Strategic Program Review. Life Sciences did receive
some feedback during the Review and via the BNL Deputy Director for
Science and Technology following the Review. Life Sciences Directorate
managers expressed a need to receive timely detailed feedback from the
Strategic Program Review and other external reviews. IO previously
reported similar concerns emanating from the High Energy and Nuclear
Physics Directorate. As noted in this report, BNL personnel expended
considerable time and effort in preparing for the Strategic Program
Review. BNL senior management should ensure that timely feedback is
provided to BNL Directorates when such reviews or evaluations are
conducted. \[Note: A draft report was received by Life Sciences after
interviews were completed.\]

[]{#__RefHeading___Toc68064789 .anchor}Exhibit 1 - Documents Reviewed

1.  Life Sciences Directorate FY03 Self Assessment Plan, Rev. 1
    (04/25/03)

2.  Life Sciences Directorate FY04 Self Assessment Plan, Rev. 0
    (12/14/03)

3.  Memorandum from H. Benveniste to Medical department Staff, Medical
    Department Leadership Changes (10/06/03)

4.  Presentation: Life Sciences Operations - Management Review
    (September 22, 2003)

5.  Memorandum from A. Emrick to Distribution, Life Sciences EMS &
    Operations Management Review Minutes (September 23, 2003)

6.  Presentation by C. Anderson, DOE Strategic Review of Biology
    (2/11/04)

7.  Presentation by H. Benveniste, Overview and Vision: Medical &
    Imaging Sciences (2/12/04)

8.  Report of the Visiting Committee for the Life Sciences Directorate,
    November 15-17, 2000

9.  \[R&R Group\] Assessment Tracking (for non-ATS issues) Updated
    1/26/04

10. \[Isotope Production\] Performance Measures Report, December 2003

11. Isotope Comment/Complaint Form \[Attachment to TPL Manual Document
    TPL 1004, Distribution of Radionuclides, Rev. 4\] (2/12/03)

12. TPL Manual Document TPL 1008 Rev 1, cGMP Assessment Program
    (4/03/02)

13. Memorandum from J. Bullis to L. Mausner, Routine cGMP Assessment
    (9/19/03)

14. Memorandum from J. Bullis to L. Mausner, Report: Routine cGMP
    Assessment (1/06/03)

15. Memorandum from J. Bullis to G. Shepherd, Building 801 Radiological
    Inventory Control Assessment (9/18/03)

16. Memorandum from J. Bullis/S. Moss to A. Emrick, Building 801
    Radiological Inventory Control Assessment (8/13/02)

17. NSRL-1 Run, Final Report (January 2004)

18. Memorandum from J. Terry to Quality Assurance, Care & Safety
    Committee, Meeting Agenda for Thursday, February 19, 2004 (Revised
    2/17/04)

19. Minutes of the Quality Assurance, Care & Safety Committee, Clinical
    Research Center, January 5, 2004

20. CRC Protocol Compliance Monitoring Records (4) (3/15/03 -- 1/16/04)

21. Memorandum from J. Terry to QAC&S Committee, Detailed Summary of
    Subject Surveys rated other than 'Good' for the period of 10/1/03 --
    12/31/03 (1/09/04)

22. Memorandum from J. Terry to QAC&S Committee, Detailed Summary of
    Subject Surveys rated other than 'Good' for the period of 7/1/03 --
    9/30/03 (1/07/04)

23. Memorandum from J. Terry to A. Baumann, Subject Follow-up for period
    10/1/03 -- 12/31/03 (2/12/04)

24. E-mail from J. Terry to C. Harris, FW: MMC Work Order (12/12/03)

25. Clinical Research Center Subject Identification Form (QAC&SC
    Approved 7/03/02)

26. Joint Commission on Accreditation of Healthcare Organizations
    \[Website\], Periodic Performance Review for Brookhaven National
    Laboratory (Printed 2/20/04)

27. General Clinical Research Centers (GCRC) National Annual Meeting,
    Association of Administrative Directors, Tentative Agenda (Printed
    2/20/04)

28. Memorandum from P. Freimuth to Distribution, Safety Inspection
    (2/22/04)

29. Memorandum from R. Colichio to L. Mausner, 2004 1^st^ Qtr Safety
    Inspection -- Bldg. 801 (2/24/04)

30. Memoranda (7) from R. Colichio to \[various\], 2004 1^st^ Qtr Safety
    Inspection (2/24/04)

31. OSHA Corrective Action Status Reports \[Buildings 463 & 801\]
    (12/11/03)

32. Worker Safety and Health Required Assessment Aid (8/20/03)

33. Radiation Survey Techniques Assessment Record (9/10/03)

34. Use of Dosimetry Assessment Record (12/16/03)

35. Operational Readiness Evaluation \[Action Status\], Building 490
    Room 9-802, Lab Renovation (2/15/02)

36. Summary of All Biology Projects \[Cost/Budget Table\] (2/29/04)

Exhibit 2 -- Personnel Interviewed

1.  H. Benveniste, Interim ALD for Life Sciences

2.  C. Anderson, Chair, Biology Department

3.  W. Gunther, Special Assistant to the ALD for Life Sciences

4.  A. Emrick, Self-Assessment Manager

5.  J. Bullis, Radionuclide & Radiopharmaceutical Isotope Production

6.  R. Colichio, ES&H Manager

7.  J. Desmond, Business Operations Manager

8.  J. Rowan, Manager, Clinical Research Center

9.  L. Stiegler, Facility Support Representative

10. M. Vazquez, NASA Space Radiation Laboratory

Exhibit 3 - Documents Referenced

1.  SBMS Management System Description: Integrated Assessment Program,
    Issue Date: July 1999

2.  BNL Independent Oversight Office FY 2004 Program Plan: Review of
    Organizational Self-Assessment Programs (December 30, 2003)

3.  U.S. Department of Energy Contract with Brookhaven Science
    Associates, DE-AC02-98CH10886

4.  Appendix B (to DE-AC02-98CH10886), Critical Outcomes, Objectives,
    and Performance Measures FY 2004 Brookhaven National Laboratory
    (Effective 03/16/04)

5.  Year End Self-Evaluation - Brookhaven National Laboratory - Fiscal
    Year 2003 (December 19, 2003)

6.  Brookhaven National Laboratory Draft Institutional Plan FY 2004 --
    FY 2008, September 2003

7.  SBMS Subject Area: Integrated Assessment, Effective Date: January
    2004

8.  SBMS ESH Standard 1.2.1, Corrective Action Management and Tracking
    for External and Internal Assessments, Rev. 5, Effective Date:
    December 2002

9.  Brookhaven National Laboratory Independent Oversight Report SA
    01-05, Evaluation of the Self-Assessment Program of the Life
    Sciences Directorate (September 26, 2001)

10. Brookhaven National Laboratory Independent Assessment Report IO
    03-12, Process for Use of Controlled Substances in Research at BNL
    (October 31, 2003)

11. 21 CFR 211, Current Good Manufacturing Practice for Finished
    Pharmaceuticals