## TABLE OF CONTENTS

1.0 Introduction 2

2.0 Evaluation Criteria 5

3.0 Evaluation Process 15

4.0 Third Party Review Team Procedures 21

5.0 BAO and BNL Roles in Process 24

6.0 References 2 6

**ACRONYMS**

BAO Brookhaven Area Office

BNL Brookhaven National Laboratory

DOE Department of Energy

R2A2s Roles, Responsibilities, Authorities, and Accountabilities

**LIST OF FIGURES AND TABLES**

Figure 1 -- BNL Integrated Assessment Program 2

Figure 2 -- BNL Assessment Processes 3

Figure 3 -- Management System Assessment Cycle 4

Figure 4 -- Conceptual Model of Evaluation Process 15

Table 1 -- Definition of Rating Categories 17

Table 2 -- Value Point Scale 19

Table 3 -- List of Key Activities 23

1.0 INTRODUCTION

**1.1 Background**

The objectives of the Integrated Assessment Program at Brookhaven
National Laboratory (BNL) are (1) to systematically and comprehensively
evaluate the performance of both organizations and management systems in
their achievement of performance objectives, (2) to use the resulting
information to establish and implement a prioritized improvement agenda,
and (3) to provide valuable insights that support strategic and
operational planning. The key elements of the BNL Integrated Assessment
Program are depicted in Figure 1, which displays the essential and
on-going relationships among strategic planning, establishing objectives
and measures of performance, conducting assessment activities,
evaluating overall performance, and providing input into next generation
planning activities. Specific assessment processes used to obtain
information on performance -- which are shown in Figure 2 -- operate in
two distinct dimensions. In the vertical dimension, organizational
managers are responsible for developing and implementing plans --
including objectives and performance measures -- and evaluating progress
towards achievement of those plans. In the horizontal or programmatic
dimension, BNL Management System Stewards are responsible for planning
the progression of their management system and periodically conducting
assessments to determine maturity, adequacy, and effectiveness of key
processes administered and services provided through their system.
Independent assessment activities can evaluate performance of
organizations (vertical) or programs (horizontal).

**Figure 1. BNL Integrated Assessment Program**

**Figure 2. BNL Assessment Processes**

BNL has, for several years, been assessing the vertical dimension or its
organizational performance by conducting line self-assessments and,
through the Independent Oversight Office, has been evaluating and
improving the effectiveness of line organization self-assessment
activities. More recently, the Laboratory has begun to focus on
management system performance. This increased emphasis was initiated
through the conduct of several management system maturity evaluations,
performed in the manner of a peer review process.

In fiscal year 2003, the Laboratory plans to assess the performance of
11 of its 33 management systems and has developed a process to guide the
conduct of these activities (see Figure 3). The 11 management systems to
be assessed are Acquisition Management, Emergency Preparedness,
Environmental Management, Facility Operations, Facility Safety,
Hazardous Material Transportation, Life Cycle Asset Management,
Radiological Control, Safeguards and Security, Work Planning and
Control, and Worker Safety and Health. The remaining management systems
will be systematically included -- enabling all management systems to be
operating in an integrated performance planning and assessment cycle by
2005. Recognizing the importance of these assessments and the basic
contents of the assessment process to the long-term performance of the
Laboratory, BNL and the Brookhaven Area Office (BAO) have established a
contract performance measure for fiscal year 2003, which calls for the
review of the effectiveness of the overall assessment process using an
independent entity (or Third Party Review Team).

**1.2 Purpose, Objectives, and Organization of Protocol**

The objectives of this Protocol are to establish those attributes or
characteristics that are associated with or define a "best-in-class"
self-assessment program for management systems and to provide a
framework for determining -- using the Third Party Review Team -- to
what extent the Laboratory's management system self-assessment processes
exhibit those attributes. In order for a Third Party Review Team to
conduct its activities in a structured and formalized manner, the
processes that the Team uses must be established and understood in
advance. Accordingly, the purposes of this document (the Protocol) are
as follows:

-   To define the criteria that the Third Party Review Team will use to
    evaluate the effectiveness of the Laboratory's self-assessment
    process for management systems,

-   To describe the process that the Third Party Review Team will use to
    establish overall ratings based on application of these criteria,

-   To indicate the various activities that the Third Party Review Team
    will perform as part of review planning, review conduct, and report
    development,

-   To formalize expectations regarding interactions between the Third
    Party Review Team and BNL and BAO, and

-   To outline the BNL and BAO roles and responsibilities.

In addressing these objectives, this Protocol is organized as follows:

-   Section 2.0 outlines the specific criteria to be used (organized
    under the overall construct of Approach-Deployment-Results), key
    references, and sample lines of inquiry.

-   Section 3.0 describes the evaluation process and ratings of
    performance.

-   Section 4.0 discusses Third Party Review Team procedures including
    Team Leader and Team Member responsibilities, review planning and
    conduct, and report development

-   Section 5.0 addresses expectations for BNL and BAO in the overall
    process.

**Figure 3. Management System Assessment Cycle**

**2.0 EVALUATION CRITERIA**

The attributes of a "best-in-class" self-assessment program are
established in the form of criterion statements. The bases upon which
these criterion statements have been developed are discussed in Section
2.1, and the criterion statements are provided in Section 2.2 along with
key references and lines of inquiry to be used by the Third Party Review
Team. Determining the extent to which these attributes are found to
exist (i.e., the criterion statements are met) in the BNL management
system self-assessment program will be accomplished via detailed
applications of the key references and sample lines of inquiry for each
criterion statement as implemented by the evaluation process described
in Section 3.

**2.1 Overview**

A number of foundation documents from a variety of settings provided
critical inputs to the development of the criterion statements presented
in Section 2.2. These documents -- which address the concepts of
organizational performance, self-assessment, work processes, information
management, and organizational and process improvements -- include the
following:

-   Department of Commerce, Baldridge National Quality Program,
    *Criteria for Performance Excellence*, 2003,

```{=html}
<!-- -->
```
-   Department of Energy (DOE) Order 414.1A, *Quality Assurance,*

```{=html}
<!-- -->
```
-   DOE Guide 414.1-2*, Quality Assurance Management System Guide,*

```{=html}
<!-- -->
```
-   DOE Policy 450.5, *Line Environment, Safety and Health Oversight,*

```{=html}
<!-- -->
```
-   Institute for Nuclear Power Operations, Principles for Effective
    Self-Assessment and Corrective Actions Programs, December 1999, and

```{=html}
<!-- -->
```
-   10 CFR 830, *Nuclear Safety Management*.

The full range of concepts outlined in these documents was considered in
establishing the criterion statements in Section 2.2. The criteria
presented reflect those specific concepts that both BAO and BNL believe
to be most relevant and important in establishing a "best-in-class"
self-assessment program at the Laboratory.

It is also essential that the criterion statements accurately reflect
the principles and objectives of the Laboratory's Integrated Assessment
Program. These principles and objectives include:

-   Providing operational, technical, and business performance
    information to effectively and efficiently monitor progress towards
    achieving strategic and high priority objectives (e.g., those having
    significant risk and/or vulnerability) and to promote early
    identification of problems that may impact their achievement;

-   Providing operational, technical, and business performance
    information to ensure levels of excellence that have been achieved
    in key areas of the Laboratory are maintained;

-   Contributing to improved organizational and management system
    performance and driving the Laboratory\'s improvement agenda by
    identifying on a prioritized basis strategic and tactical elements
    for next-generation Critical Outcomes, Objectives, and Performance
    Measures;

-   Verifying expectations and requirements of stakeholders and
    customers to improve scientific and technological research, improve
    the delivery of support services and products, and enhance customer
    satisfaction;

-   Involving managers and staff commensurate with their level of
    responsibility and accountability; and

```{=html}
<!-- -->
```
-   Verifying conformance to established internal and external
    requirements.

Collectively, the essence of these principles and objectives of the
Laboratory's Integrated Assessment Program are embodied in the 10
criterion statements -- outlined below -- for evaluating management
system self-assessment effectiveness. Each criterion statement is
supported by a series of key reference documents and sample lines of
inquiry to be used by the Third Party Review Team in performing its
evaluation. The criterion statements are presented under the construct
of Approach-Deployment-Results, which are defined as "dimensions" in the
Malcolm Balridge process and are used to allow for effective measurement
of organizational and system performance and for identification of
improvement initiatives. This construct recognizes that initiating,
effecting, and measuring change is an evolutionary process that must be
systematically planned, implemented, and evaluated. Overarching criteria
for each of the dimensions of Approach-Deployment-Results are also
provided in Section 2.2.

**2.2 Criterion Statements, Key References, and Sample Lines of
Inquiry**

**APPROACH: Is the management system self-assessment program
comprehensively defined and does the structure of the program support
adequate planning activities that enable effective measurement of
performance?**

**Criterion 1 -- The Program is Comprehensively Defined and Effectively
Communicated.**

The Laboratory has established a strategic vision, associated
objectives, and expectations for the management system self-assessment
program, which provides the framework for formalized processes and
implementing procedures -- including its relationship to the Integrated
Planning Management System and the Integrated Assessment Program -- and
has effectively communicated the vision, objectives, expectations, and
process framework to the line organizations and Management System
Stewards.

**Key References:**

-   Laboratory Institutional Plan

-   Critical Outcomes, Objectives, and Performance Measures (Appendix B)

-   Integrated Planning Management System Description

-   Integrated Assessment Program Management System Description

-   Integrated Assessment Subject Area

-   Quality Management System Description

-   Quality Program Description

**Sample Lines of Inquiry:**

1.  Is the management system self-assessment process linked to
    Laboratory strategic documents, including but not limited to the
    Laboratory Institutional Plan, Appendix B of the contract, which
    contains the Critical Outcome Trees (critical outcomes, objectives,
    and performance measures), and other contractual requirements?

2.  Do the Integrated Assessment Program Management System Description
    and the Integrated Assessment Subject Area clearly describe the
    vision, objectives, and expectations for the Laboratory's management
    systems and the management system self-assessment process?

3.  Have the strategic vision, associated objectives, and expectations
    been effectively communicated throughout the organization?

4.  How has this been accomplished?

**Criterion 2 -- There is Institutional Commitment to and Broad
Acceptance of the Program.**

Throughout the Laboratory, support for the management system
self-assessment program is clearly evident, consistently reinforced, and
reflected in decision-making, assignment of priorities, and allocation
of resources. The line organization managers, Management System
Stewards, points of contact, subject matter experts, and working level
personnel understand and accept the strategic vision, associated
objectives and expectations, and the formalized process and implementing
procedures for the management system self-assessment program.

**Key References:**

-   Critical Outcomes, Objectives, and Performance Measures (Appendix B)

-   Integrated Assessment Program Management System Description

-   Integrated Assessment Subject Area

-   Organizational Business and Operating Plans

-   Budget Process, Organizational Budgets, 3PBP

-   Management System Self-Assessment Program Plan(s) and Report(s)

-   Information Management Tools and Technology

-   Assessment Tracking System

# 

Sample Lines of Inquiry:

1.  Does management demonstrate its commitment to the management system
    self-assessment program through assuring the availability of
    essential systems and human resources and reinforcing the importance
    of effectively planning and executing self-assessment activities?

2.  Is the management system self-assessment process documented as an
    essential activity in organizational business and operating plans?

3.  Is line management's commitment to the management system
    self-assessment process evident to working level personnel?

4.  How has this been demonstrated and is it consistently reinforced?

5.  Is the process for allocating resources to management system
    self-assessment activities well-understood, and are resources
    assigned to these self-assessment activities?

6.  Is there an annual, prioritized budget for the management system
    self-assessment program?

7.  Are there discretionary resources available if and when needed?

8.  Are critical information management tools and technology readily
    available?

9.  Are essential implementing systems readily available and functional?

10. Do line organization managers, staff, Management System Stewards,
    points of contact, and subject matter experts accept and understand
    the Integrated Assessment Program Management System Description and
    Subject Area documents (including procedures) as implementing
    guidance for conducting management system self-assessment
    activities?

11. Do the line organization managers and staff and the Management
    System Stewards, points of contact, and subject matter experts
    accept and understand the guidance contained in the flow chart
    outlining the management system self-assessment program (see Figure
    3)?

12. Do management system self-assessment program plans demonstrate an
    understanding of the fundamental principles of the management system
    self-assessment process?

**Criterion 3 -- Essential Planning Elements and Processes are
Systemmatic in Their Design and Application.**

Development of management system self-assessment plans considers all
aspects or functions (including overall program effectiveness) but
focuses on high priority issues and/or activities (e.g., those having
significant risk and/or vulnerability); assessment techniques are
intended to be graded or tailored to the function being evaluated; input
from customer and stakeholders is solicited and considered in
development of self-assessment programs; and pre-established performance
measures or expectations are used to determine management system
effectiveness.

**Key References:**

-   Management System Self-Assessment Program Plan(s)

-   Management System Objectives and Performance Measures

```{=html}
<!-- -->
```
-   Assessment Schedules, Scope of Activities, and Supporting Milestones

-   Annual Management System Review

-   Integrated Assessment Subject Area

-   Laboratory Institutional Plan

-   Critical Outcomes, Objectives, and Performance Measures (Appendix B)

**Sample Lines of Inquiry:**

1.  Does the Management System Self-Assessment Program Plan indicate
    that all functions associated with the operation of a management
    system are, on a prioritized basis, included as part of the
    potential self-assessment scope?

2.  Are the performance objectives/measures and metrics aligned with the
    strategic objectives and improvement agenda for the management
    system and does achievement of management system performance
    objectives support achievement of Laboratory Critical Outcomes and
    other priority initiatives?

3.  Are contractual and/or regulatory-based drivers used to scope and
    design assessment activities?

4.  Are high priority and high strategic impact elements or issues
    considered in scoping and designing assessment activities?

5.  Are the management system performance objectives, performance
    measures, and metrics used to scope, structure, and guide management
    system self-assessment activities?

6.  Are management system performance objectives/measures established,
    understood, accepted, and designed to measure or indicate increasing
    maturity of the management system?

7.  Do the customers involved in self-assessment activities reflect the
    range of conditions under which the management system operates
    (organizations, type of activity, and risk profile)?

8.  Is there a logical process to prioritize what functions will be
    assessed, which incorporates notions of past performance?

9.  Are plans and schedules for conducting self-assessment developed
    and/or formalized and are these plans provided to stakeholders,
    customers, and line management?

10. Is an annual review of the entire program part of the scope and are
    the individual management system self-assessment activities designed
    (at least in part) to support this overall program evaluation?

11. Do the Integrated Assessment Program Management System Description
    and subordinate Subject Area documentation (including procedures)
    provide sufficient guidance on how to implement management system
    self-assessment activities?

12. Is the guidance provided consistent with the flow chart that has
    been developed for the management system self-assessment program
    (see Figure 3 in this report)?

13. Are management system self-assessment program plans consistent with
    the overall guidance contained in the Integrated Assessment Program
    Management System Description and Subject Area documentation
    (including procedures)?

14. Does the self-assessment program plan indicate the intent to employ
    -- as appropriate -- a variety of assessment techniques including
    work observations, inspections, peer reviews (e.g., management
    system maturity evaluations, experimental reviews), walk-through(s),
    and table top analysis?

15. Is the need for the use of independent assessment activities
    considered in the design of the self-assessment process?

16. Does the self-assessment plan indicate the intent to use customers
    and stakeholders in the execution of self-assessment activities?

**Criterion 4 -- Roles and Responsibilities are Established and
Personnel Competency is Demonstrated.**

Roles, Responsibilities, Authorities, and Accountabilities (R2A2s) for
conducting management system self-assessment activities are assigned,
documented, and well understood. Clear relationships exist among
management system self-assessment activities conducted by Management
System Stewards, points of contact, and subject matter experts; line
organization managers and staff; and internal independent organizations.
Personnel with appropriate expertise/technical competency conduct
management system self-assessment activities; all personnel are
encouraged to provide information on management system effectiveness.

**Key References:**

-   Management System Self-Assessment Program Plan(s)

```{=html}
<!-- -->
```
-   R2A2s for senior Laboratory managers, organizational managers,
    staff, management system stewards, points of contact, and subject
    matter experts.

```{=html}
<!-- -->
```
-   Independent Oversight Office Program Plan

-   Internal Audit Program Plan

-   Job Training Assessments

```{=html}
<!-- -->
```
-   Training/Mentoring Records

```{=html}
<!-- -->
```
-   Employee Feedback Procedures and Processes

**Sample Lines of Inquiry:**

1.  Are the roles and responsibilities identified in the Integrated
    Assessment Program Management System Description consistent with
    R2A2 documentation for line managers and staff and Management System
    Stewards, points of contact, and subject matter experts?

2.  Are the assessment-specific R2A2s in the Management System
    Self-Assessment Program Plan consistent with other R2A2
    documentation?

3.  Do the Management System Self-Assessment Program Plan and/or
    underlying implementation plans explicitly identify who will be
    conducting self-assessment activities?

4.  Are clear references made to related/supporting assessments to be
    conducted by other organizational elements?

5.  Is there a clear understanding of the self-assessment activities to
    be conducted by the Management System Steward versus those to be
    conducted by either the line organization or other Laboratory
    elements? Is this understanding formalized?

6.  Are personnel trained or otherwise receiving mentoring on these
    guidelines?

7.  Are personnel allowed to observe self-assessment activities as a
    training mechanism?

8.  Are personnel encouraged to provide information on management system
    effectiveness?

**DEPLOYMENT: Is the management system self-assessment program
implemented as designed, and does the self-assessment program
effectively measure management system performance?**

**Criterion 5 -- Self-Assessment Program Plan(s) are Executed as
Designed and Modified as Circumstances Warrant.**

Management system self-assessment activities are executed and documented
as planned. If conditions or priorities change (e.g., unexpected
performance, resource availability, and external factors), the
self-assessment program is sufficiently flexible to allow for
re-examination of the planning baseline and re-prioritization of
specific self-assessment activities as necessary.

**Key References:**

-   Schedule Versus Actual Completion of Assessment Activities

-   Integrated Assessment Subject Area

-   Subordinate Procedures and Protocols

-   Assessment Reports

**Sample Lines of Inquiry:**

1.  Are self-assessment schedules and plans consistently followed during
    execution?

2.  Are procedures and protocols for conduct of self-assessment
    activities followed?

3.  Are pre-established change control processes used when assessment
    schedules are adjusted?

4.  Are appropriate managers, customers, and stakeholders informed when
    assessment priorities change?

5.  Are appropriate techniques and methods used to conduct
    self-assessments (e.g., on-going, periodic, informal, and formal)?

6.  Is work observation used to evaluate the extent to which fundamental
    work planning and control processes are being properly performed?

7.  Are peer review processes used when special expertise is needed or
    when it is important to demonstrate credibility of the process?

8.  Has the self-assessment process demonstrated sufficient flexibility
    when conditions or assessment priorities changed?

**Criterion 6 -- Assessment Objectives are Achieved and Results are
Communicated to Affected Stakeholders.**

Management system self-assessment activities produce outcomes aligned
with the assessment objectives, yield documented measures of performance
(e.g., findings, strengths, weaknesses, improvement opportunities, and
noteworthy practices), and are clearly communicated to and understood by
stakeholders impacted by the assessment results (e.g., Management System
Stewards, line organization) -- including an appropriate level of
disclosure to DOE and regulatory agencies based on significance and in
accordance with contractual obligations.

**Key References:**

-   Self-Assessment Plans

-   Self-Assessment Reports

-   Records of Communication and Disclosure

-   Meeting Minutes, Presentations, and Attendance Records

-   Lessons Learned

-   Annual Program Review

**Sample Lines of Inquiry:**

1.  Do management system self-assessments focus on performance outcomes
    or processes?

2.  Are self-assessment results logically derived from the analytical
    information presented?

3.  Is there a correlation between self-assessment results and
    self-assessment objectives -- that is, was the correct information
    obtained?

4.  Are self-assessment results in a form that addresses performance and
    enables establishment of corrective actions or improvement
    initiatives (i.e., are outcomes readily actionable)?

5.  Are self-assessment results communicated clearly, in a timely
    manner, and understood?

6.  What mechanisms are used to communicate self-assessment results?

7.  To whom are results communicated? Does this include stakeholders,
    DOE, and regulators?

8.  Are self-assessment results documented and/or recorded?

9.  Is a lessons learned program used to capture and share information
    relevant to the Laboratory as a whole or across the DOE complex?

10. Is there evidence that the program considers lessons derived from
    performance weaknesses as well as from noteworthy practices?

11. Do the management system self-assessment results serve also as input
    to the overall evaluation of the entire program?

12. Does senior management consider the need to inform the Brookhaven
    Science Associates Board of self-assessment results? How is such a
    decision made?

**Criterion 7 -- Results are Analyzed to Determine the Existence of
Management System and Institutional Performance Trends.**

Assessment results are analyzed from management system and, as
appropriate, organizational perspectives; analysis of assessment results
includes determining the underlying reason(s) for either strong
performance or identified weaknesses, which guides the development and
implementation of appropriate actions for sustaining management systems
performance or preventing recurrence or performance deficiencies.
Assessment results are analyzed, as appropriate, for the identification
of potential programmatic deficiencies and/or the effectiveness of past
Management System Steward or line organization manager actions.

**Key References:**

-   Analysis of Performance Information

-   Trending and Analysis Process

-   Self-Assessment Reports

**Sample Lines of Inquiry:**

1.  

2.  

3.  What types of analysis processes are used and are they consistent
    with Laboratory requirements and expectations?

4.  What type(s) of events or conditions have been the subject of causal
    factor analysis (i.e., is it limited to only environment, safety,
    and health aspects)?

5.  Is this process being used as input to trending and analysis
    activities?

6.  Is trending and analysis of past and recent events performed to
    determine patterns in performance, identify areas of potential
    vulnerability, and focus out-year planning?

7.  Are trending and analysis activities limited to the identification
    of management system performance weaknesses?

8.  Does this process allow for integration of management system
    performance information and determination of systemic performance
    issues?

9.  Does this process enable identification of high strategic impact
    items (e.g., having wide application across the organization)?

10. Is this information used to develop an overall assessment of
    management system self- assessment program health or performance?

11. Is self-assessment program performance compared against expected
    levels of performance?

12. Are benchmarks used and, if so, how?

13. Does analysis confirm the relationship between defined and
    implemented management systems and observed results?

**Criterion 8 -- An Established Process of Action Planning, Tracking,
Closure, and Verification Exists and Operates Effectively.**

Management system and organizational conditions and actions are
prioritized; high priority conditions and actions (e.g., those having
significant risk and/or vulnerability) are considered for inclusion as
critical outcome objectives or performance measures, or as priority
initiatives in other planning processes. Resource needs are considered
in budget development (and/or change control) processes and resources
are committed commensurate with prioritization. R2A2s are established;
agreed upon actions (corrective, preventive, or interim) are implemented
and tracked to closure in a formal reporting system; and action
effectiveness is evaluated, as appropriate.

**Key References:**

-   Assessment Tracking System or Similar System(s)

-   ESH Standard 1.2.1

-   Causal Analysis and Corrective Action Subject Area

```{=html}
<!-- -->
```
-   Budget Process, Organizational Budgets, and 3PBP

-   Laboratory Institutional Plan

**Sample Lines of Inquiry:**

1.  How were conditions and actions prioritized?

2.  How have resource needs been considered and committed?

3.  Is a formal system used to manage and track performance findings?

4.  Are actions completed as scheduled? Are revisions to scheduled
    completion dates performed in a timely manner and based on a
    balanced consideration of priorities?

5.  Is explicit responsibility and accountability assigned to an
    individual (e.g., Management System Steward or his designee) for the
    resolution of each corrective action?

6.  How have the assessment owners ensured the adequacy of actions
    taken?

# RESULTS: Are the results from the management system self-assessment process effectively used to improve operational performance, both in the activities of the management system and in the self-assessment program itself?

**Criterion 9 -- Conduct of Self-Assessment Activities Leads to
Improvements in Operational Performance and Achievement of Near and
Longer-Term Objectives.**

Results of management system self-assessment activities are used by
Management System Stewards to support achievement of near-term and
longer-term objectives and to ultimately drive improvements in the
performance of their management systems.

**Key References:**

-   Lessons Learned

-   Self-Assessment Reports

-   External Assessment Reports

-   Independent Assessment Reports

-   Annual Program Review

```{=html}
<!-- -->
```
-   Improvement Agendas

-   Strategic, Business, and Operational Plans

```{=html}
<!-- -->
```
-   Changes to Standards Based Management System Documents

**Sample Lines of Inquiry:**

1.  Are self-assessments leading or contributing to improvements in
    operational performance?

2.  Are self-assessments leading or contributing to improvements in
    performance beyond the specific scope of the assessment area?

3.  Has this process reduced programmatic deficiencies?

4.  Are self-assessments leading or contributing to achievement of
    near-term and longer-term management system objectives?

5.  Is this information used to drive strategic planning activities and
    operational planning?

6.  Does this information contribute to development of improvement
    agendas?

Criterion 10 -- The Program Leads to Continuous Improvement in
Assessment Processes and is Valued by Customers and Stakeholders.

The results of management system self-assessments -- including
stakeholder and customer feedback -- are used to structure and
prioritize future, management system self-assessment activities and
contribute to improvements in assessment processes. The management
system self-assessment program is clearly recognized as providing high
strategic value to both customers and stakeholders.

Key References:

-   Stakeholder and Customer Feedback Records

```{=html}
<!-- -->
```
-   Business and Operational Plans

```{=html}
<!-- -->
```
-   Self-Assessment Plans

-   Self-Assessment Reports

-   Annual Program Review

-   Correlation of Frequency of Self-Assessment Activities with Prior
    Performance

-   Out-year Assessment Schedules and/or Objectives

Sample Lines of Inquiry:

1.  Are analyses of self-assessment activities and associated events
    performed and used to focus out-year planning?

2.  Is this information used as input to establishing the frequency and
    nature of future self-assessment activities?

3.  Is this information used to improve the conduct of the
    self-assessment activities?

4.  How has self-assessment planning and implementation improved?

5.  Is the self-assessment program evident to stakeholders and
    customers?

6.  Do stakeholders and customers value the self-assessment program?

7.  Is stakeholder and customer feedback sought in designing future
    self-assessment activities?

8.  Is the solicitation rigorous and integrated with the strategy for
    conduct of self-assessment activities?

9.  How has this input been used to establish improvement initiatives
    and drive improvements in management system performance?

**3.0 EVALUATION PROCESS**

The process that the Third Party Review Team will employ to evaluate the
effectiveness of the BNL management system self-assessment program is
described in this section. The objective of the evaluation process is to
provide a systematic, comprehensive, and traceable mechanism for
determining the overall program performance of the BNL management system
self-assessment program. The process consists of the following elements:
(1) adjectivally rate the extent to which each criterion statement is
being achieved or satisfied for the set of management system
self-assessment processes examined (Section 3.1); (2) develop an
aggregate adjectival rating for the overall self-assessment dimensions
of Approach, Deployment, and Results (Section 3.1); (3) convert the
aggregate adjectival rating to a numerical rating for each dimension
(Section 3.2); and (4) determine an overall composite program rating
(Section 3.3). The concepts to be described are depicted in Figure 4.

+---------+-------------------+-------------------+-------------------+
| **Man   | **Program         |                   |                   |
| agement | Dimensions and    |                   |                   |
| S       | Criteria**        |                   |                   |
| ystem** |                   |                   |                   |
+---------+-------------------+-------------------+-------------------+
|         | **Approach        | **Deployment      | **Results         |
|         | Criteria**        | Criteria**        | Criteria**        |
+---------+-------------------+-------------------+-------------------+
|         | **Criteria 1 --   | **Criteria 5 --   | **Criteria 9 --   |
|         | 4**               | 8**               | 10**              |
+---------+-------------------+-------------------+-------------------+
| **MS    |                   |                   |                   |
| 1**     |                   |                   |                   |
+---------+-------------------+-------------------+-------------------+
| **MS    |                   |                   |                   |
| 2**     |                   |                   |                   |
+---------+-------------------+-------------------+-------------------+
| **MS    |                   |                   |                   |
| ...**   |                   |                   |                   |
+---------+-------------------+-------------------+-------------------+
| **MS    |                   |                   |                   |
| ...**   |                   |                   |                   |
+---------+-------------------+-------------------+-------------------+
| **MS    |                   |                   |                   |
| n**     |                   |                   |                   |
+---------+-------------------+-------------------+-------------------+
| *       | **Overall         | **Overall         | **Overall         |
| *Degree | Adjectival Rating | Adjectival Rating | Adjectival Rating |
| of      | for Each of       | for Each of       | for Each of       |
| Achiev  | Criteria 1 -- 4** | Criteria 5 -- 8** | Criteria 9 --     |
| ement** |                   |                   | 10**              |
|         |                   |                   |                   |
| **Of    |                   |                   |                   |
| Cri     |                   |                   |                   |
| teria** |                   |                   |                   |
+---------+-------------------+-------------------+-------------------+
| *       | **Overall         | **Overall         | **Overall         |
| *Rating | Adjectival**      | Adjectival**      | Adjectival**      |
| for     |                   |                   |                   |
| Dime    | **Approach        | **Deployment      | **Results         |
| nsion** | Rating**          | Rating**          | Rating**          |
+---------+-------------------+-------------------+-------------------+

**Figure 4. Conceptual Model of Evaluation Process**

**3.1 Evaluate Criterion Statements and Program Activities**

The initial step in the evaluation process is for the Third Party Review
Team to develop a rating of the extent to which each of the 10 criteria
is being achieved or satisfied by the management system self-assessment
activities that are being examined. This is accomplished by the use of a
set of rating categories or judgments. For this application, the rating
categories of "Outstanding," "Excellent," "Good," "Marginal," and
"Unsatisfactory" will be used. These categories are consistent with the
adjectival ratings that are used by BAO to evaluate the overall
Laboratory performance of Brookhaven Science Associates. Definitions for
each of these rating categories -- as applied to the management system
self-assessment process -- are provided in Table 1.

Accordingly, each of the criteria that are Approach-related are assigned
a rating based on the Third Party Review Team's collective judgment
regarding which rating category best characterizes the performance of
the Laboratory on that criteria (using the Table 1 definitions). This
process is similarly repeated for all Deployment-related and
Results-related criteria -- yielding 10 individual criterion rating
judgments across all the management systems self-assessment processes
being examined. Once these individual criterion judgments have been
established, the Third Party Review Team then develops an aggregate
adjectival rating (again using the Table 1 definitions) for each of the
dimensions -- Approach, Deployment, and Results -- by integrating the
individual judgments within each of these three dimensions. This yields
three adjectival rating judgments.

**3.2 Convert Aggregate Adjectival Ratings to Numerical Rating**

This step of the evaluation process involves translating the adjectival
ratings for each dimension to a numerical rating. This conversion is
accomplished with the aid of the value point scale provided in Table 2.
The Third Party Review Team will assign a two-digit numerical rating to
each dimension area using the ranges provided in Table 2 as a guide. For
example, if the Review Team were to determine that the dimension of
Approach should receive an adjectival rating of "Excellent," then the
assigned numerical rating would need to be selected within the range of
greater than 2.5 and less than or equal to 3.5 (e.g., 2.6). This step
yields three numerical rating scores -- one for Approach, Deployment,
and Results. In addition, the Review Team should provide the rationale
or distinguishing factors that lead to the assignment of each of the
three ratings within the overall numerical ranges available. Using the
example above, the Review Team would document why the dimension of
Approach was assigned a numerical rating of 2.6 versus a rating that
could be as potentially favorable as 3.5. These documented judgments
will be essential to developing an integrated set of Review Team
recommendations (see Section 3.4).

+-----------+----------------------------------------------------------+
| *         |                                                          |
| *Table 1. |                                                          |
| D         |                                                          |
| efinition |                                                          |
| of Rating |                                                          |
| C         |                                                          |
| ategories |                                                          |
| for       |                                                          |
| Self-A    |                                                          |
| ssessment |                                                          |
| Act       |                                                          |
| ivities** |                                                          |
+===========+==========================================================+
| **        | **Approach**                                             |
| Ratings** |                                                          |
+-----------+----------------------------------------------------------+
| **Outs    | -   An effective, systematic self-assessment approach,   |
| tanding** |     for the institution and management systems, that is  |
|           |     fully responsive to the requirements of the          |
|           |     management system and the current and changing needs |
|           |     is evident                                           |
|           |                                                          |
|           | -   The self-assessment approach is fully integrated     |
|           |     with the management system and organizational needs  |
+-----------+----------------------------------------------------------+
| **Ex      | -   An effective, systematic self-assessment approach,   |
| cellent** |     for the institution and management systems, that is  |
|           |     responsive to most requirements of the management    |
|           |     system and most current and changing needs is        |
|           |     evident                                              |
|           |                                                          |
|           | -   The self-assessment approach is well integrated the  |
|           |     management system and organizational needs           |
+-----------+----------------------------------------------------------+
| **Good**  | -   An effective, systematic self-assessment approach,   |
|           |     for the institution and management systems, that is  |
|           |     responsive to many requirements of the management    |
|           |     system and key current and changing business needs   |
|           |     is evident                                           |
|           |                                                          |
|           | -   The self-assessment approach is aligned with basic   |
|           |     management system and organizational needs           |
+-----------+----------------------------------------------------------+
| **M       | -   The beginnings of a systemic self-assessment         |
| arginal** |     approach, for the institution and management         |
|           |     systems, that is responsive to the basic             |
|           |     requirements of the management system is evident     |
|           |                                                          |
|           | -   Early stages of a transition from reacting to        |
|           |     problems to a general improvement orientation are    |
|           |     evident                                              |
+-----------+----------------------------------------------------------+
| **Unsatis | -   There is limited evidence of a systematic approach   |
| factory** |                                                          |
|           | -   Reaction to problems is the dominant ethic           |
+-----------+----------------------------------------------------------+
|           | **Deployment**                                           |
+-----------+----------------------------------------------------------+
| **Outs    | -   The self-assessment approach is fully deployed       |
| tanding** |     without significant weaknesses or gaps in any areas  |
|           |     of the management system                             |
|           |                                                          |
|           | -   A very strong, fact-based systemic evaluation and    |
|           |     improvement process and extensive learning/sharing   |
|           |     of management system performance information are key |
|           |     management tools                                     |
|           |                                                          |
|           | -   Strong refinement, innovation, and integration --    |
|           |     backed by excellent management system level analysis |
|           |     and sharing -- exist                                 |
+-----------+----------------------------------------------------------+
| **Ex      | -   The self-assessment approach is well deployed with   |
| cellent** |     no significant gaps                                  |
|           |                                                          |
|           | -   A fact-based systemic evaluation and improvement     |
|           |     process and learning/sharing of management system    |
|           |     performance information are key management tools     |
|           |                                                          |
|           | -   Evidence exists of refinement, innovation, and       |
|           |     improved integration as a result of management       |
|           |     system level analysis and sharing                    |
+-----------+----------------------------------------------------------+
| **Good**  | -   The self-assessment approach is well deployed,       |
|           |     although deployment may vary in some areas of the    |
|           |     management system                                    |
|           |                                                          |
|           | -   A fact-based systemic evaluation and improvement     |
|           |     process is in place for improving the efficiency and |
|           |     effectiveness of key elements of the management      |
|           |     system                                               |
+-----------+----------------------------------------------------------+
| **M       | -   Major gaps exist in deployment; some areas are in    |
| arginal** |     the early stages of deployment                       |
|           |                                                          |
|           | -   These gaps inhibit progress in achieving basic       |
|           |     management system requirements                       |
+-----------+----------------------------------------------------------+
| **Unsatis | -   There is limited evidence of deployment              |
| factory** |                                                          |
|           | -   Performance information is largely anecdotal in      |
|           |     nature and does not contribute to achieving basic    |
|           |     management system requirements                       |
+-----------+----------------------------------------------------------+
|           | **Results**                                              |
+-----------+----------------------------------------------------------+
| **Outs    | -   Deployment of the self-assessment process is leading |
| tanding** |     to excellent improvement trends **and/or** sustained |
|           |     outstanding performance levels in most areas of      |
|           |     importance to the management system                  |
|           |                                                          |
|           | -   The rigor of the self-assessment process and         |
|           |     performance of the management system demonstrate     |
|           |     evidence of industry **and** benchmark leadership in |
|           |     many areas                                           |
|           |                                                          |
|           | -   Self-assessment results fully address key            |
|           |     customer/stakeholder, Management System Steward, and |
|           |     improvement agenda requirements                      |
+-----------+----------------------------------------------------------+
| **Ex      | -   Deployment of the self-assessment process is leading |
| cellent** |     to sustained improvement trends **and/or** good to   |
|           |     excellent performance levels in areas of importance  |
|           |     to the management system                             |
|           |                                                          |
|           | -   The rigor of the self-assessment process and         |
|           |     performance of the management system -- when         |
|           |     evaluated against relevant comparisons **and/or**    |
|           |     benchmarks -- show many areas of leadership and very |
|           |     good relative performance levels                     |
|           |                                                          |
|           | -   Self-assessment results address most key             |
|           |     customer/stakeholder, Management System Steward, and |
|           |     improvement agenda requirements                      |
+-----------+----------------------------------------------------------+
| **Good**  | -   Deployment of the self-assessment process is leading |
|           |     to improvement trends **and/or** good performance    |
|           |     levels in most areas of importance to the management |
|           |     system                                               |
|           |                                                          |
|           | -   The rigor of the self-assessment process and         |
|           |     performance of the management system -- when         |
|           |     evaluated against relevant comparisons **and/or**    |
|           |     benchmarks -- show some areas of strength and good   |
|           |     to very good relative performance levels             |
|           |                                                          |
|           | ```{=html}                                               |
|           | <!-- -->                                                 |
|           | ```                                                      |
|           | -   Self-assessment results address most key             |
|           |     customer/stakeholder and Management System Steward   |
|           |     requirements                                         |
+-----------+----------------------------------------------------------+
| **M       | -   Deployment of the self-assessment process is leading |
| arginal** |     to some improvements **and/or** early good           |
|           |     performance levels in a few areas of the management  |
|           |     system                                               |
|           |                                                          |
|           | -   Self-assessment results do not address many to most  |
|           |     key customer/stakeholder, Management System Steward, |
|           |     and improvement agenda requirements                  |
+-----------+----------------------------------------------------------+
| **Unsatis | -   Performance levels and trends are less than          |
| factory** |     satisfactory in most areas of the management system  |
|           |                                                          |
|           | -   Self-assessment results -- to the extent that they   |
|           |     exist -- do not address most key                     |
|           |     customer/stakeholder, Management System Steward, and |
|           |     improvement agenda requirements                      |
+-----------+----------------------------------------------------------+

**Table** **2. Value Point Scale**

  -------------------------- --------------------------------------------
  **Rating**                 **Rating Score**

  **Outstanding**            **Greater than 3.5 and less than or equal to
                             4.0**

  **Excellent**              **Greater than 2.5 and less than or equal to
                             3.5**

  **Good**                   **Greater than 1.5 and less than or equal to
                             2.5**

  **Marginal**               **Greater than 0.5 and less than or equal to
                             1.5**

  **Unsatisfactory**         **Less than or equal to 0.5**
  -------------------------- --------------------------------------------

**3.3 Determine Overall Program Rating**

The final step in evaluation process is to translate the numerical
ratings for each of the dimensions into a single composite rating for
overall program performance. Completion of this step involves the
assignment of "weights" or measures of relative importance to each of
the three dimensions. For example, if the Review Team were to determine
that each of the three dimensions were to receive equal weighting (i.e.,
all were judged to be equal in relative importance), then the overall
program rating would be established by averaging the three numerical
ratings arrived at from the previous step of the evaluation process.
Conversely, if the Third Party Review Team were to determine that one of
the dimensions was relatively more important to the notion of overall
program performance than the others, then weighting factors would be
assigned to each dimension prior to integrate the three individual
ratings into a composite rating. When assigning the relative weights,
the Review Team may consider, as they deem appropriate, the state of
maturation of the management system.

By way of example, if the Review Team were to apply weights based on the
dimension of Approach being judged "twice as important" as Deployment or
Results and numerical ratings had been assigned as 3.6 for Approach, 3.2
for Deployment, and 2.4 for Results, the following composite rating
would result:

**Composite Rating** =

(Approach Relative Weight) (Approach Numerical Rating) + (Deployment
Relative Weight) (Deployment Numerical Rating) + (Results Relative
Weight) (Results Numerical Rating)

**Composite Rating** =

(0.5)(3.6) + (0.25)(3.2) + (0.25)(2.4) = 3.2

**3.4 Identify Improvement Initiatives for Self-Assessment Program**

It will be essential for the Third Party Review Team to provide a basis
for its judgments and conclusions in a form that can be readily
interpreted by the Laboratory and subsequently translated into
improvement actions (should the ratings be consistent with the need to
implement such). Identification of Improvement Initiatives is essential
if the Laboratory is to achieve its strategic objective of a
"best-in-class" self-assessment program.

Accordingly, based on its findings in evaluating the 10 criteria, and
assigning both adjectival and ultimately numerial ratings, the Review
Team will formulate potential recommendations for Laboratory management
on improvements to the BNL management system self-assessment program.
These recommendations will consider the management systems
self-assessment processes reviewed as well as Laboratory-wide processes
that support effective execution of management systems self-assessment
activities. Recommendations will be linked to components of the
management system self-assessment process and must be actionable in that
they can be readily translated into corrective actions, which can be
assigned by the Laboratory to a discrete party and tracked to closure
using metrics for measuring progress.

**4.0 THIRD PARTY REVIEW TEAM PROCEDURES**

For the Third Party Review to be successful, it must be credible.
Credibility demonstration requires a number of elements, including but
not limited to, the credentials of the Team Leader and members, the
existence of well-defined roles and responsibilities for such, a
well-conceived and articulated work flow process (i.e., review planning,
including formal evaluation criteria, review conduct, report
development), and effective communication. These concepts are discussed
in this section.

**4.1 Team Composition and Qualifications**

There are a number of important qualifications that should be inherent
in the Team Leader, the Team members, and the entire Review Team
collectively. First, it is highly desirable that the Team Leader (1) has
demonstrated experience in the design, implementation, and evaluation of
self-assessment and performance improvement processes; (2) has served in
a leadership capacity on multidisciplinary assessment teams; (3)
understands the national laboratory system and the processes for
assessing performance; and (4) is familiar with the mission of
Brookhaven National Laboratory. Second, it is highly desirable that the
Team members (1) understand the essential elements of highly effective
and mature self-assessment and performance improvement processes; (2)
have conducted assessment activities measuring the effectiveness and
maturity of organizational and system performance; and (3) are familiar
with the national laboratory system and the mission of Brookhaven
National Laboratory. Third, the entire Review Team should have -- among
the collective expertise of its members -- (1) current or previous
management responsibility for organizations and/or programs; (2)
experience in research and development environments; (3) knowledge of
environment, safety, health, and quality aspects; (4) background in
operational settings; and (4) familiarity with business support
functions. Finally, Team members must demonstrate objectivity,
professionalism, and credibility throughout the process.

**4.2 Team Leader Responsibilities**

The Team Leader is the primary interface with the Laboratory on matters
of review content and provides overall direction to Team Members.
Critical planning and operational responsibilities of the Team Leader
include the following:

-   Determining assignments (focus areas) for each of the Team members
    (e.g., Team members could be assigned specific management systems
    and/or specific criteria to evaluate), communicating these
    assignments, and monitoring progress of Team activities,

-   Developing, in concert with the entire Team, the ratings for each
    criterion statement (aggregate ratings by
    Approach-Deployment-Results, and ratings for the overall program),

-   Managing end-of-day and other team meetings,

-   Expeditiously conveying to the Laboratory the need for additional
    documents or interviews,

-   Ensuring that the judgments (ratings) accurately and appropriately
    reflect the majority views of the entire Team,

-   Coordinating presentation of the review out-briefing,

-   Overseeing development of a draft and final report, and

-   In cooperation with the Laboratory, ensuring that logistical
    arrangements and support functions meet the needs of the Team.

**4.3 Team Member Responsibilities**

It is essential that Team members work in collaborative and cohesive
fashion to ensure that there is a consensus achieved as the review
results are developed. Team member responsibilities include the
following:

-   Conducting evaluation scope assigned to them by the Team Leader (see
    first bullet in 4.2),

-   Participating in establishing a rating for each criterion statement
    (aggregate ratings by Approach-Deployment-Results, and ratings for
    the overall program),

-   Participating in daily and other Team meetings,

-   Proactively identifying additional documents and/or interviews that
    may be necessary to support their review scope,

-   Ensuring that their insights and judgments are accurately reflected,

-   Participating in the review out-briefing, and

-   Developing information for the review report.

**4.4 Review Planning**

Review planning is essential to ensure that expectations among Team
members are identified and well-understood. Review planning includes the
following activities:

-   Developing a strategy for the allocation of the scope of the review
    among the Team members (e.g., Team members work in pairs,
    individually, or as a group),

-   Designing the strategy for interactions with the Laboratory and BAO
    (e.g., nature and frequency of interactions, request for
    information, length of onsite visit, front-end briefings),

-   Developing an information request for hard copy and electronically
    provided information,

-   Developing an initial list of potential interviewees, and

-   Ensuring that provisions are in place for addressing logistical
    issues (office, computers, phone, site contacts).

Most of the above elements will be documented in a Work Plan to be
provided by the Review Team to the Laboratory roughly one month in
advance of the onsite visit. The Work Plan will both guide the efforts
of the Team and assist the Laboratory in preparing for the onsite visit.
The Work Plan should address elements not specifically identified in
this Protocol; these include the specific scope of the Third Party
Review, the roles of Team members in terms of the execution of the
scope, specific interactions planned with the Laboratory and their
timing, prospective interviewees, key documents necessary to review as
background information, and a proposed schedule of planning, execution,
and report development activities (see Section 4.7).

**4.5 Review Implementation**

Review implementation includes any activity in which the Team is
interacting with the Laboratory and BAO for purposes of obtaining an
understanding of the management system self-assessment process and
evaluating the effectiveness of that process. Review implementation
includes the following activities:

-   Participating in an in-briefing with Laboratory and BAO personnel to
    describe the expected activities of the Team and to receive
    presentations on the each of the management system self-assessment
    processes, if such have not been provided previously,

-   Conducting document reviews, interviews, and other reviews outlined
    in the Work Plan,

-   Conducting Team-only meetings daily to review Team member
    observations and recommendations and examine the subsequent day's
    plan,

-   Communicating periodically with the Laboratory during onsite
    activities,

-   Providing an out-briefing to Laboratory and BAO personnel on review
    findings, and

-   Determining an appropriate schedule for preparation of the draft and
    final reports and conveying these to the Laboratory (see Section
    4.7).

**4.6 Report Development**

Report development involves ensuring that a clear and concise
description of review results is provided in a timely fashion and
includes candidate improvement items (should identification of such be
warranted by the ratings). Report development includes the following
activities:

-   Developing an annotated outline,

-   Identifying writing assignments for preparation of initial draft
    input,

-   Working with a Facilitator to ensure timely coordination of inputs
    and development of a first draft report (see Section 5.1),

-   Ensuring that the draft report is reviewed by the Team,

-   Transmitting the report to the Laboratory for Factual Accuracy
    Review,

-   Understanding, evaluating, and incorporating comments on the Factual
    Accuracy draft, and

-   Developing a final report.

**4.7 Assessment Schedule**

To ensure that conduct of the Third Party Review is performed in an
organized and systematic manner, it is suggested that the following
schedule serve as the guide to the timing of specific activities. The
dates in Table 3 are in terms of time before or after the onsite visit.

## 

## Table 3. Schedule of Key Activities

+----------------------------------------+-----------------------------+
| ## Activity                            | **Time Prior/After Onsite   |
|                                        | Visit**                     |
+----------------------------------------+-----------------------------+
| 1.  Finalize Protocol                  | 3 months prior              |
+----------------------------------------+-----------------------------+
| 2.  Establish Team                     | 3 months prior              |
+----------------------------------------+-----------------------------+
| 3.  Provide Work Plan to Laboratory    | 1 month prior               |
+----------------------------------------+-----------------------------+
| 4.  Schedule Interviews                | 2 weeks prior               |
+----------------------------------------+-----------------------------+
| 5.  Confirm Final Logistics for Review | 1 week prior                |
+----------------------------------------+-----------------------------+
| 6.  Provide Formal Out-briefing        | At the end of onsite visit  |
|     Indicating Preliminary Results of  |                             |
|     Review                             |                             |
+----------------------------------------+-----------------------------+
| 7.  Provide BNL/BAO with Factual       | At the end of onsite visit  |
|     Accuracy Report                    |                             |
+----------------------------------------+-----------------------------+
| 8.  Receive BNL/BAO Factual Accuracy   | 1 weeks after               |
|     Comments                           |                             |
+----------------------------------------+-----------------------------+
| 9.  Provide BNL/BAO with Final Report  | 2 weeks after               |
+----------------------------------------+-----------------------------+

**5.0 BNL AND BAO ROLES IN PROCESS**

Both the Laboratory and BAO have vital roles in contributing to the
success of the Third Party Review. These roles involve participation in
activities associated with review planning, review execution, and report
development.

**5.1 BNL Roles and Expectations**

The Laboratory will be interacting with the Third Party Review Team on a
number of levels. These include: identifying potential members to the
Team Leader for consideration, furnishing access to background
information, providing briefings on key aspects of each of the
management systems and the associated self-assessment activities,
coordinating requests for interviews, providing other logistical support
during onsite activities, and providing resources to facilitate
development of the Review Team report.

-   **Establishment of the Third Party Review Protocol.** In partnership
    with BAO, the Laboratory will develop and finalize the Protocol
    Document, including criterion statements, key references, and sample
    lines of inquiry, that will guide the Third Party Review Team

-   **Identification of Potential Team Members.** The Laboratory will
    provide the Team Leader with names of potential candidates that the
    Team Leader may wish to consider and/or identify the types of
    expertise/qualifications that the Team should exhibit to ensure
    credibility (see Section 4.1).

-   **Access to Background Information.** The Third Party Review Team,
    in conjunction with the Laboratory, will develop a list of key
    documents. The Laboratory will be responsible for providing these to
    the Team (whether in hard copy or electronic form) in a timely
    manner.

-   **Presentations on Management Systems.** Each of the Management
    System Stewards -- whose system will be part of the review scope --
    will be providing a detailed briefing for the Third Party Review
    Team. These briefings will describe the system, the approach to the
    self-assessment process for the particular system, the status of
    implementation of assessment activities, and resulting process
    improvement activities.

-   **Coordination of Interviews.** Once the Third Party Review Team has
    established an initial interview list, the Laboratory will be
    responsible for arranging all interviews as well as ensuring that
    all changes to the master interview list are managed and expedited.

-   **Logistical Support.** The Laboratory will designate a
    point-of-contact to facilitate all logistical support while the Team
    is onsite. This includes site access, office space, conference
    rooms, telephones and fax, computer support, and office supplies.

-   **Provision of Resources to Facilitate Report Development.** The
    Laboratory will also designate a point-of-contact (or Facilitator)
    who will support the Third Party Review Team in developing the draft
    (Factual Accuracy) and final reports; this is envisioned to include
    technical writing, editing, compilation of inputs, and production.

-   **Review of Draft Report for Factual Accuracy.** The Laboratory will
    be responsible for reviewing the draft report for factual accuracy
    in a timely manner. The nature of this review is to identify if
    there are factual errors or omissions in the document. The review is
    not to debate issues of interpretation or conclusion. After receipt
    of comments, the Third Party Review Team will correct any factual
    errors and will determine, if any such factual errors are
    identified, whether they affect the conclusions in the report.

**5.2 BAO Roles and Expectations**

BAO has four potential roles in this process: supporting establishment
of the protocol, furnishing relevant information, providing
presentations on BAO activities, and participating as interviewees in
the interview process.

-   **Establishment of the Third Party Review Protocol.** In partnership
    with the Laboratory, BAO will develop and finalize the Protocol
    Document -- including criterion statements, key references, and
    sample lines of inquiry -- that will guide the Third Party Review
    Team

```{=html}
<!-- -->
```
-   **Access to Background Information.** BAO may be asked by the Third
    Party Review Team to furnish hard copy or electronic information on
    specific assessments of interest.

-   **Presentation on Relevant BAO Activities.** The Team may find it
    advantageous for BAO to present information on its oversight role
    and on assessment activities that it has conducted (which involve
    examination of the management systems of interest).

-   **Participating as Interviewees in the Interview Process.** The Team
    may determine that it is desirable to interview BAO personnel as the
    Team develops its understanding of the BNL management system
    self-assessment process.

**6.0 REFERENCES**

10 CFR 830.120, *Nuclear Safety Management*

Brookhaven National Laboratory, DH-SOP-005, *Planning Institutional
Independent Assessments*, May 1, 2002

Brookhaven National Laboratory, *Integrated Assessment Program
Management System Description*, July 1999

Department of Commerce, Baldridge National Quality Program, *Criteria
for Performance Excellence*, 2003

DOE G 414.1-1, *Implementation Guide for use with Independent and
Management Assessment Requirements of 10 CFR Part 830.120 and DOE
5700.6C Quality Assurance*, August 1996

DOE G 414.1-2, *Quality Assurance Management System Guide for use with
10 CFR 830.120 and DOE O 414.1*, June 17, 1999

DOE O 414, 1A, *Quality Assurance*

DOE P 450.5, *Line Management, Safety and Health Oversight*

Institute for Nuclear Power Operations, *Principles for Effective
Self-Assessment and Corrective Actions Programs*, December 1999

  -------- --------------------------------------------------------------
           

           

           

           

           
  -------- --------------------------------------------------------------

**Table 3. Definition of Rating Categories (continued)**

  ------------- -----------------------------------------------------------
  **Ratings**   **Deployment**

                

                

                

                
  ------------- -----------------------------------------------------------

**Table 3. Definition of Rating Categories (continued)**

  ------------- -----------------------------------------------------------
  **Ratings**   **Results**

                

                

                

                
  ------------- -----------------------------------------------------------