Brookhaven Science Associates

Brookhaven National Laboratory

Independent Oversight Report

SA 03-01

Review of Self-Assessment Program

of the

Community, Education, Government,

and Public Affairs Directorate

July 8, 2003

Prepared by:

John Usher, Reviewer Date

> Independent Oversight Office

Approved by:

Robert McNair, Manager Date

Independent Oversight Office

OS 40-03

Contents

1.0 Introduction [1](#introduction)

1.1 Purpose and Scope [1](#purpose-and-scope)

1.2 Review Process [1](#review-process)

1.3 Review Criteria [2](#review-criteria)

2.0 Program Summary [3](#program-summary)

2.1 Criterion 1: Approach to Self-Assessment
[3](#criterion-1-approach-to-self-assessment)

2.2 Criterion 2: Deployment of Self-Assessment
[6](#criterion-2-deployment-of-self-assessment)

2.3 Criterion 3: Results from Self-Assessment
[8](#criterion-3-results-from-self-assessment)

3.0 Conclusions [10](#conclusions)

3.1 Strengths [10](#strengths)

3.2 Areas for Improvement [11](#areas-for-improvement)

3.3 IAP Management System Programmatic Issues
[12](#iap-management-system-programmatic-issues)

**Exhibits**

Exhibit 1 - Documents Reviewed [13](#__RefHeading___Toc38940660)

Exhibit 2 -- Personnel Interviewed [14](#__RefHeading___Toc38940662)

Exhibit 3 - Documents Referenced [15](#__RefHeading___Toc38940663)

## Introduction

### Purpose and Scope

This report presents the results of Independent Oversight's review of
the Self-Assessment (SA) Program of the Community, Education,
Government, and Public Affairs (CEGPA) Directorate. As an element of the
Integrated Assessment Program (IAP), the Independent Oversight (IO)
Office is chartered to independently verify the effectiveness of
Brookhaven National Laboratory's Self-Assessment Program. To accomplish
this, the IO Office performs a review of Directorates or other
organizations reporting to the Laboratory Director in accordance with
the requirements and guidelines provided in the IAP Management System
Description. Reviews are identified and scheduled in the FY 2003 Program
Plan: Review of Organizational Self-Assessment Programs \[FY03 IO Review
Plan\].

FY03 reviews focus on the organization's "approach" to self-assessment,
the status of "deployment" and the "use of results" to improve
performance. Specifically, IO will review how the organization is
planning to implement the self-assessment program, how the organization
conducts the self-assessment program, and how the results from
self-assessment are analyzed and used to improve performance. In the
course of this review, comparison of the current status of the CEGPA SA
Program to that reported in **Independent Oversight Report SA 00-06,
Evaluation of the Community Involvement, Government, and Public Affairs
Directorate Self-Assessment Program \[IO Report SA 00-06\]** was used to
measure progress since the FY 2000 evaluation.

### Review Process

This review formally began on February 24, 2003, with an inbriefing
conducted with personnel from CEGPA. This meeting served to introduce
the IO Reviewer, John Usher, who provided information on review
objectives, methodology, review criteria, and schedule. The inbriefing
also served to inform the IO Reviewer about CEGPA and to identify key
personnel involved in the planning and implementation of
self-assessment.

The following data and collection methods were used during the review:

-   Document Reviews (Exhibit 1)

```{=html}
<!-- -->
```
-   Interviews (Exhibit 2)

```{=html}
<!-- -->
```
-   Documents Referenced (Exhibit 3)

Interviews were conducted using prepared questions tailored to address
the review criteria (see Section 1.3) in terms of key business
factors/performance objectives. Notes were prepared during each
interview to record the information gathered. Comments, concerns, and
follow‑up actions were also documented after each interview.

An outbriefing was conducted on June 9, 2003, with personnel from CEGPA.
At this meeting, the IO Reviewer summarized results of the review and
submitted a draft report for review by CEGPA. The meeting also gave
CEGPA personnel the opportunity to provide feedback to the IO Reviewer
on the review process.

### Review Criteria 

The IO assessment process evaluates an organization's self-assessment
program against the criteria published in the **FY03 IO Review Plan**.
These criteria are:

1.  Approach

```{=html}
<!-- -->
```
1.  The scopes of planned assessment activities are comprehensive, have
    a strong technical basis, and are balanced with work activities.

2.  The methods for conducting key scheduled assessments are defined and
    are commensurate with types of assessments planned and performance
    information desired.

3.  Management and staff involvement commensurate with their
    responsibilities is evident.

4.  Key supporting organizational processes (tracking/trending systems,
    causal analysis, critiques, etc.) and tools are developed.

5.  Organizational plans/procedures address regulatory and/or SBMS
    drivers for assessment activities.

```{=html}
<!-- -->
```
2.  Deployment

 

1.  Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.

2.  Assessments are documented and communicated as planned.

3.  Assessment results are evaluated/analyzed to a degree commensurate
    with the type of assessment. Strengths are identified.
    Corrective/improvement actions are identified, prioritized, assigned
    to specific owners, and assigned due dates. All conditions are
    tracked to closure.

4.  Evidence of timely self-identification of issues exists. Significant
    issues are brought to the attention of management and disclosed to
    regulatory/oversight agencies in accordance with contractual
    obligations.

5.  Management involvement is evident.

```{=html}
<!-- -->
```
3.  Results

```{=html}
<!-- -->
```
1.  Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations, such as mission
    achievement and retention/expansion of core competencies.

2.  Evidence exists that there is an appropriate connection between
    results of organizational self-evaluation and development of
    strategic/institutional plans.

Further detailed information supplementing the review criteria is
provided in the **FY03 IO Review Plan**
(<http://www.io.bnl.gov/safy03.doc>).

## Program Summary

This review covers the CEGPA Directorate comprising the Community
Involvement Office, the Media and Communications Office, and the Office
of Educational Programs (OEP). The Assistant Laboratory Director (ALD)
for CEGPA is the Steward of three SBMS Management Systems: External
Communications, Internal Communications, and Education Programs.

A discussion and analysis of data collected for the review of the
Directorate is presented for each criterion. Areas that demonstrated
positive performance or programmatic strengths (criteria exceeded) are
identified with a (+). Areas where criteria are met but not exceeded are
unmarked. Areas that indicated opportunities for improvement (criteria
not met or partially met) are noted using a (-).

### Criterion 1: Approach to Self-Assessment

The Special Assistant to the ALD CEGPA prepared the CEGPA Directorate
Self Assessment Plan. The Plan was approved by the ALD on February 24,
2003.

The scope of the SA program is comprehensive in addressing the
Directorate's operations, ES&H, business/finance, and human resources.
Managers select appropriate methodologies for assessment activities,
although these methodologies are not always described in the SA Plan. It
is evident that activities are appropriately prioritized. Managers,
staff, and key stakeholders are appropriately involved in planning
assessment activities. Processes for supporting the organizational SA
programs are in place, although Directorate personnel do not use the
Family Assessment Tracking System(s) available to them. Evidence exists
that organizational managers appropriately consider SBMS and contractual
drivers in developing performance goals/objectives and in planning and
scheduling assessment activities, although not all BNL required
assessments, as added in January 2003, relevant to CEGPA were
acknowledged in the SA Plan.

1.  **(+) The scopes of planned assessment activities are comprehensive,
    have a strong technical basis, and are balanced with work
    activities.**

Assessment activities are prioritized based on alignment with BNL and
organizational strategic plans, results of previous assessments,
business and operational risk, and available resources. The SA Plan is
clearly aligned with the Institutional Plan and the BNL Prime Contract,
Appendix B. The SA Plan reflects results of previous assessments,
especially the Communications and Trust Advisory Panel (CTAP) peer
review.

(+) The SA Plan is based on a framework that ensures a comprehensive
perspective of organizational performance. The SA Plan is organized
based on Critical Outcome 3.5 along with the Environmental Management
System; Environment, Safety, and Health; Training and Qualifications;
Financial Reviews; Operational Performance; and Diversity. The SA Plan
clearly covers the full scope of the organization's operational,
business, and financial activities. The SA Plan specifically
incorporates objectives and measures, under Operational Performance, for
Routine Operations, High School Program, Lab Link, Civic Outreach
Program, Summer Sundays, support provided to Environmental Restoration
Projects, and Targeting the "Science Attentive" and "Science Interested"
Public.

The SA Plan evidences appropriate attention to customer
requirements/expectations. Stakeholder input and feedback is key in
development of performance measures. DOE customer expectations are
reflected in Critical Outcomes and in the documentation from the DOE
Office of Workforce Development for Teachers and Scientists.

(+) Progress on, and effectiveness of, actions in response to the annual
CTAP review are included in the scope of the next CTAP review and are
appropriately considered in subsequent planning. Critiques conducted
following CEGPA events or activities provide an additional source of
responses that are appropriately considered in planning.

Results from external assessments are incorporated into the
organization's self-assessment and annual self-evaluation process. The
SA Plan incorporates results from CTAP peer review and DOE Office of
Workforce Development for Teachers and Scientists on-site review of
BNL's educational programs.

External lessons learned are incorporated into the organization's
self-assessment process. CEGPA organizations incorporate lessons learned
through the CTAP process, DOE's program review, and interaction with
other peer groups at meetings, conferences, and conventions.

2.  **(-) The methods for conducting key scheduled assessments are
    defined and are commensurate with types of assessments planned and
    performance information desired.**

Assessment activities are assigned to qualified personnel including
appropriate use of subject matter experts. Personnel well versed in the
areas being assessed conduct self-assessment activities. Appropriately
qualified personnel conduct Environmental Assessments and Tier I
inspections. CTAP comprises subject matter experts who evaluate CEGPA's
performance relative to Critical Outcomes.

Managers effectively communicate their expectations such that all
employees understand their roles and responsibilities for
self-assessment. Management expectations for self-assessment are
documented in the SA Plan and in the CEGPA Critical Outcome Tracking
spreadsheet. Additionally, roles and responsibilities are communicated
to CEGPA personnel through manager and employee performance goals and
objectives and R2A2s.

Managers ensure that independent assessments and peer reviews are
considered and incorporated into the self-assessment process, as
appropriate. CEGPA combines independent assessment and peer review
through the CTAP process. Internal peer review is incorporated through
presentation dry runs and performance observations.

CEGPA uses an appropriate variety of information sources in collecting
information relevant to organizational performance. Approaches include
one-on-one contacts, stakeholder surveys, focus groups, analysis of
feedback obtained via e-mail, staff and management meetings, analysis of
demographic data, analysis of data reflecting website usage, review of
training records, review of budgets and expenditures, post-event
critiques, walkthroughs, assessments, presentation dry runs, and
performance observations.

(-) The SA Plan does not specifically describe the approaches (methods
used to gather performance information) to be used for several areas
listed under Operational Performance. Under High School Program, Lab
Link, Civic Outreach Program, Summer Sundays, and Environmental
Management, the relevant paragraphs contain generic statements that the
items will be assessed, reviewed, and/or evaluated without providing
detail as to the specific approach(es) to be used.

Assessment activities do not typically impact the work of other
organizations or other personnel. Many assessment activities are
integrated into the normal performance of the work being measured, such
as focus groups reviewing their performance or survey forms distributed
during performance. Assessment activities such as Tier I inspections or
Environmental Assessments are planned to involve the personnel in the
work areas being assessed to include them in the assessment process.

3.  **Management and staff involvement commensurate with their
    responsibilities is evident.**

Planning involves managers and staff members as appropriate to ensure
that all aspects of the organization's operations are evaluated. The
Special Assistant to the ALD solicited input from managers during the
development of the SA Plan. Managers were also involved in the
development of Critical Outcome 3.5 as an ongoing part of the CTAP
process designed to define objectives for the coming fiscal year.
Managers solicit feedback regarding performance objectives from staff
through staff meetings, the performance appraisal and individual goal
planning process, and informal contacts.

(-) It is noted that "update of the organization\'s self-assessment
program for the current year" was not completed by December 31 per the
SBMS Subject Area: Integrated Assessment. This was in large part due to
the delay in approval of Appendix B to the BNL Contract for FY03. The
ALD CEGPA and Level 2 Managers reviewed and approved the SA Plan
(2/24/03).

4.  **(+) Key supporting organizational processes (tracking/trending
    systems, causal analysis, critiques, etc.) and tools are
    developed.**

(+) The organization has processes in place for analyzing and trending
the results of assessment activities as well as organizational and
Laboratory performance. The information package and/or the management
presentations provided to CTAP and for the DOE On-site Program Review of
OEP contain evidence (documentation) of assessment results and analysis
of results. Notably, the Community Involvement Office prepares CEGPA
Decision Papers to document assessment results and issues analyses. The
Issues Management Database provides a tool not only for
planning/evaluating CEGPA activities and performance but also for
planning and evaluating BNL-wide performance.

High-level actions (i.e., those specifically related to Critical Outcome
3.5) are tracked on the CEGPA Critical Outcome Tracking spreadsheet.
Other actions are tracked using manager-specific processes such as MS
Outlook Task Manager, organization-specific databases, and personal
planners or notebooks. Action status is reviewed at management and staff
meetings and is documented to some extent in the CEGPA Weekly Report
Forecast, which is distributed throughout the Directorate. Some Tier I
actions were tracked in the Director's Office Family ATS. Those related
to the Environmental Management (EM) Program are tracked in the EM
Family ATS. CEGPA has three Family (Community Involvement, Educational
Programs, and Media & Communications) ATS available but does not use
these tracking systems. It is noted that many actions accomplished by
CEGPA personnel require such short turnaround that these do not merit
entry into any sort of tracking system or process.

(+) Available information systems are used to effectively support
assessment activities. Notably, the Web Trends statistical tool,
provided through the Information Technology Division, is used to support
data analysis regarding the usage of CEGPA websites. It is also noted
that OEP is working with Human Resources to improve analytical
capabilities for demographic data (PeopleSoft modules).

5.  **(-) Organizational plans/procedures address regulatory and/or SBMS
    drivers for assessment activities.**

There is clear, explicit linkage to Critical Outcome 3.5 documented in
the SA Plan, Section 1.0 Critical Outcomes and Performance Measures, as
well as the CEGPA Critical Outcome Tracking spreadsheet.

(-) Other objectives and performance measures are not as clearly
documented. While an overriding statement that notes the contribution of
the areas listed under Operational Performance to the Critical Outcomes
and Performance Measures is included, specific performance objectives
for each area assessed are not clearly described. Several of the
paragraphs begin with a brief history of the service or product but
without specifically stating the objective to be achieved.

(-) Some required assessments (Tier I inspections and Environmental
Assessments) were incorporated into the SA Plan. The Special Assistant
to the ALD acknowledged that he had not yet incorporated required
assessments newly listed in the Required Self-assessments exhibit
(revision issued January 2003) from the SBMS Subject Area: Integrated
Assessment due to the timing for preparation of the SA Plan.

### Criterion 2: Deployment of Self-Assessment

Assessment activities are conducted as scheduled. Schedules are adjusted
based on changing priorities as determined by organizational managers.
Results of assessment activities are documented and communicated to
responsible CEGPA managers for review and analysis. Key assessment
results are regularly discussed and analyzed at management and/or staff
meetings. Managers evaluate assessment results to identify both
strengths and opportunities for improvement. Necessary improvement
actions are identified and communicated to responsible managers and
stakeholders.

1.  **Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.**

Those activities scheduled for first quarter of FY03 have been completed
as evidenced by, for example, internal decision papers and the Critical
Outcome Tracking spreadsheet. Some activities are ongoing, as opposed to
discrete assessments, and are reported periodically.

The updated SA Plan for FY03 was first issued on February 24, 2003. The
Special Assistant to the ALD stated that SA Plan would be updated to
incorporate, as appropriate, the recently issued update to the SBMS
Subject Area: Integrated Assessment regarding required assessments.

The scope and schedule of planned assessments are adjusted when
appropriate based on significant performance information or changing
priorities. Planned assessment activities are updated according to
operational circumstances; especially in response to major events taking
place at BNL, such as the awarding of a Nobel Prize to R. Davis. As
issues emerge, adjustments are made to organizational operations and
assessment activities addressing those operations.

(+) Customer feedback is almost continuously solicited as part of the
organization's assessment activities. Customer input/feedback is
solicited both before and after events through approaches such as focus
groups, roundtables, surveys, one-on-one contacts, and periodic
management and staff meetings. Customer feedback approaches are built
into many of the services provided by the Community Involvement Office
associated with planning and conduct of Environmental Restoration
Projects. Customers/stakeholders are included in post-event critiques.

2.  **Assessments are documented and communicated as planned.**

Assessment activities are documented and results are provided to
responsible managers. Evident examples include CEGPA Decision Papers,
Tier I results, Environmental Assessment results, CTAP report, and DOE's
program review report. Also, mid-year status of Critical Outcome 3.5 was
communicated to BNL and DOE management at the Annual Partners' Retreat.

The assessment schedule is managed to ensure information is available to
support timely management decisions. Evident examples include post-event
critiques and follow up on reviews of Environmental Restoration Project
documents. Much of the information provided to management is prepared to
support the annual reviews by CTAP and DOE.

3.  **Assessment results (internal and external) are evaluated/analyzed
    to a degree commensurate with the type of assessment. Strengths are
    identified. Corrective/improvement actions are identified,
    prioritized, assigned to specific owners, and assigned due dates.
    All conditions are tracked to closure.**

The organization has an effective process for identifying opportunities
for improvement. The CEGPA Decision Papers contain recommendations for
improvement. CTAP and DOE reports identify recommendations for
improvement. Managers described additional specific examples of
identified opportunities for improvement during interviews.

Improvements are prioritized in accordance with business and operational
risks and cost benefit. The ALD and Level 2 managers clearly consider
risk to BNL in prioritization including, for example, the FY03 focus on
Community Involvement processes as reflecting the importance of BNL's
Environmental Restoration Project activities. The ALD also discussed the
focus on the Internal Communications Plan as key to improving employee
communications at BNL. The availability of resources is a key driver in
prioritization for CEGPA.

Improvements are validated, as commensurate with the level of risk, to
ensure desired outcomes are achieved. Key improvements (e.g., those
affecting Critical Outcomes) are validated by CTAP in year-end
evaluation.

4.  **(+) Evidence of timely self-identification of issues exists.
    Significant issues are brought to the attention of management and
    disclosed to regulatory/oversight agencies in accordance with
    contractual obligations.**

(+) The Issues Management Database provides information on extant and
emerging issues of interest to both CEGPA and to the larger BNL
community. The Community Relations Manager provided one-on-one training
to BNL ALDs on the use of this database. BNL managers were previously
trained on the SBMS Subject Area: Community Involvement in Decision
Making.

Assessment results are analyzed to determine potential impacts on SBMS
documents. Management Systems and Subject Areas are revised as
appropriate pending results of assessment activities. It is not known at
this point if CEGPA Management Systems will be incorporated into BNL's
Management System Self-Assessment Program in FY04 or not until FY05.

Lessons learned are shared internally to CEGPA, with Environmental
Restoration Projects personnel (stakeholder communications), and with
other BNL organizations. Lessons learned are communicated to the larger
peer community both inside and outside DOE through involvement in
conferences, workshops, and peer groups including CTAP.

5.  **(+) Management involvement is evident.**

Managers participate in activities conducted or supported by the
Directorate. Self-assessment is built into many of the services,
activities, and events conducted and/or supported by CEGPA.
Self-assessments are conducted before, during, and/or after the
activities themselves. Formal plans are prepared for major
activities/events and managers participate in preparing and reviewing
these plans, which contain performance objectives, measures and
methodologies for assessments. The small size of the individual CEGPA
organizations results in managers' participation in Laboratory tours,
media events, and community involvement activities, for example.
Managers interact regularly with media representatives, regulators, and
other government officials.

(+) Managers participate in observing activities in progress (work
practices). As noted above, managers participate in work activities as
well as planning and post activity critiques (lessons learned
exercises). Where appropriate, lessons learned are communicated to
organizations outside of CEGPA so that future cross directorate
activities are improved.

Managers participate in the analysis and evaluation of assessment
results. This is primarily evident through managers' extensive knowledge
of assessment results evident through interviews and managers' conduct
of frequent meetings with staff both scheduled and informal. Managers
conduct analysis and evaluation of assessment results in order to
prepare presentations for CTAP and for DOE Program Review (OEP).

Managers verify that needed improvements are implemented. Evidence of
verification is provided in presentations prepared for CTAP, for
example.

### Criterion 3: Results from Self-Assessment

CEGPA provided input to BNL's Year End Self-Evaluation Report for FY
2002 summarizing performance results for Critical Outcome 3.5 and
identifying opportunities for improvement in FY 2003 through the CTAP
process. Critical Outcome performance reporting was accomplished via
computerized submittals to the BNL Integrated Information Management
System (IIMS). OEP prepared a document entitled, "Self-Assessment for FY
2002" evaluating performance for FY 2002 and describing strategic goals
for FY 2003.

Action plans are developed for high priority assessment activities.
Actions are appropriately prioritized and tracked to closure. Managers
review the need for changes in action due dates and approve
rescheduling. Assessment reports reviewed provide evidence of timely
self-identification of issues/concerns. Results are reviewed at
management and staff meetings. Several examples of improved operational
performance were noted throughout the Directorate. It is clear that
organizational managers are aware of vulnerabilities and areas for
improvement. There is appropriate linkage between the SA Plan and BNL's
Institutional Plan.

1.  **Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations, such as mission
    achievement and retention/expansion of core competencies.**

Progress towards achievement of near-term and long-term goals and
objectives is evident and affirmed annually by CTAP as an independent
reviewer of CEGPA's performance. CEGPA evidences maintenance of
excellence in established activities and implementation of initiatives
for improvement. CTAP verified that CEGPA's performance has been
enhanced and made more efficient as evidenced by consistency in
performance with reduction in personnel. Ample evidence of improvements
is documented in the presentations to CTAP and DOE and favorable
evaluations by CTAP and DOE provide confirmation of improvements.

Ample evidence exists that customers and other stakeholders value the
products and services provided by CEGPA. Managers receive positive
feedback (solicited and unsolicited) from BNL (Environmental Restoration
Projects and scientific Departments) and other stakeholders (media,
community, regulators, government, students, educators, etc.) as part of
assessment activities.

2.  **(+) Evidence exists that there is an appropriate connection
    between results of organizational self-evaluation and development of
    strategic/institutional plans.**

Organizational plans, such as Community Relations and Internal
Communications, are developed and improved based on input and feedback
collated as part of assessment activities. "Templates" for plans for
media events, for example, are evaluated and improved from one event to
the next as appropriate. CEGPA input to Laboratory level (strategic)
plans reflect evaluation of assessment results.

(+) Improvement actions are either acted upon at the
organization/Directorate level or communicated to BNL senior management
to be included in the Laboratory's improvement agenda, as appropriate.
Many of CEGPA's assessment activities are designed to address
Laboratory-level performance by design, and as such, results from these
activities directly impact Laboratory-level performance. Other
activities address CEGPA's performance solely or may include other BNL
organizations such as Environmental Restoration Projects. Communication
of results is evident particularly because communication is CEGPA's
primary function.

In addition to responsiveness demonstrated through the CTAP process,
CEGPA demonstrated responsiveness to other independent or external
assessments as exemplified via improvements addressing the Areas for
Improvement and the Recommendations made in **IO Report SA 00-06**:

-   **The SA Plan does not contain explicit reference to
    objectives/measures in Appendix B of the Prime Contract.** The FY03
    CEGPA SA Plan is based upon, and explicitly linked to, Critical
    Outcome 3.5, Appendix B.

```{=html}
<!-- -->
```
-   **The SA Plan does not describe all requisite elements of the SA
    process.** The FY03 CEGPA SA Plan contains a description of the SA
    "cycle" in the Introduction section of the document. With the
    exception of ES&H items, action tracking is not explicitly discussed
    in the SA plan though formalized action tracking for high-level
    actions is in place.

```{=html}
<!-- -->
```
-   **The SA Plan does not list all BNL-required self-assessment
    activities.** The FY03 CEGPA Plan explicitly includes Tier I
    inspections and Environmental Assessments but does not yet
    incorporate other required assessments per the SBMS Subject Area:
    Integrated Assessment as recently issued.

-   **The SA Plan does not describe the roles and responsibilities of
    the Deputy ALD as manager or coordinator of the self-assessment
    process.** The Special Assistant to the ALD has assumed many of the
    roles and responsibilities previously assigned to the Deputy ALD
    (departed BNL) regarding the SA Program. These are now documented in
    the R2A2s for the Special Assistant and are no longer required to be
    listed in the SA Plan.

It is noted that CEGPA made a good faith effort in responding to the
explicit recommendations from **IO Report SA 00-06**.

The CTAP process also provides linkage to establishing FY04 Critical
Outcomes. Draft FY04 Critical Outcome 3.6, Communications and Trust, was
communicated to BNL and DOE management at the Annual Partners' Retreat.
The Critical Outcome "theme" is to "demonstrate influence over external
factors that facilitate strategic Laboratory goals."

## Conclusions

### Strengths 

The CEGPA Directorate evidences improvement in the organizational
Self-Assessment Program since FY00 over what was already recognized as
an effective program. The SA Program is more formally documented and
more comprehensive in scope. Approaches to self-assessment activities
include much more than just traditional audits and represent a
performance-based concept. Roles and responsibilities for
self-assessment are clearly defined. Processes for documenting results
of activities, analyzing results, identifying opportunities for
improvement, and tracking actions are in place. The SA Plan documents an
annual cycle of continuous improvement, represents a forward-looking
approach, and provides linkage with both Critical Outcomes and the BNL
Institutional Plan in planning and achievement of longer-term goals and
objectives.

-   The CEGPA SA Program is **comprehensive in scope** encompassing
    operations/services, ES&H, business/finance, and human resources.
    The SA Plan specifically incorporates objectives and measures, under
    Operational Performance, for Routine Operations, High School
    Program, Lab Link, Civic Outreach Program, Summer Sundays, support
    provided to Environmental Restoration Projects, and Targeting the
    "Science Attentive" and "Science Interested" Public.

-   CEGPA has **processes in place for analyzing and trending** the
    results of assessment activities as well as organizational and
    Laboratory performance. Analysis of assessment results is evident in
    documentation used by CTAP and DOE's On-site Program Review. The
    Community Involvement Office prepares CEGPA Decision Papers to
    document assessment results and issues analyses. The Issues
    Management Database provides a tool for planning and evaluating both
    CEGPA and BNL-wide performance and includes emerging issues.

-   The **CTAP process is a benchmark** for peer review. The annual CTAP
    review provides the basis for evaluating BNL's performance versus
    Critical Outcome 3.5. CTAP combines peer review with independent
    review using subject matter experts thus incorporating peers'
    experiences and lesson learned. The formal CTAP review is preceded
    by CEGPA's documented self-evaluation of performance for the fiscal
    year and is succeeded by planning for the following fiscal year,
    thus ensuring a cyclic process that facilitates continuous
    improvement.

```{=html}
<!-- -->
```
-   CEGPA makes effective use of information technology in
    conducting/supporting assessment activities. The CEGPA Webmaster
    uses the Web Trends statistical tool, provided through the
    Information Technology Division, to support analysis regarding the
    usage, format and content of CEGPA websites. Additionally, OEP is
    working with Human Resources to improve analytical capabilities for
    demographic data (PeopleSoft modules).

-   CEGPA demonstrated considerable management involvement in all
    dimensions of the SA process. Managers are clearly involved in
    establishment of performance objectives, planning of assessment
    activities, conduct of activities (including performance
    observation), analysis of assessment results, and identification of
    opportunities for improvement and improvement actions.

-   CEGPA managers interviewed uniformly articulated commitment to
    improving organizational and individual performance as reflected in
    the services/products provided by CEGPA organizations. Notably,
    managers indicated a desire for further improvement even in areas
    that were highly rated by customers/stakeholders.

-   Across CEGPA, there is considerable customer and stakeholder focus.
    All organizations routinely obtain stakeholder input and feedback
    through focus groups, surveys, and one-to-one contacts. The
    development of the Internal (Employee) Communications Plan
    represents a clear milestone for both CEGPA and BNL in recognizing
    the "internal customer."

### Areas for Improvement 

-   CEGPA did not have an ALD-approved SA Plan in place by December 31.

> **Recommendation:** The ALD should ensure that FY04 SA Plan is
> prepared and approved by December 31, 2003, per the SBMS Subject Area:
> Integrated Assessment.

-   The SA Plan does not explicitly state performance objectives for
    some areas under Operational Performance.

> **Recommendation:** The ALD should ensure that each area listed in the
> SA Plan includes a clear statement as to what objective(s) is to be
> achieved. For example, an objective could be "the High School Program
> effectively provides opportunities for students and teachers to visit
> Lab facilities" or a more detailed statement. Ask what is it
> management expects the High School Program to be or to provide.

-   The SA Plan does not specifically describe the assessment approaches
    or to be used for several areas listed under Operational
    Performance.

> **Recommendation:** The ALD should ensure that assessment approaches
> or methodologies are explicitly stated in the SA Plan. For example,
> the SA Plan should state that performance versus a particular
> objective would be measured using customer surveys, focus groups, or
> one-to-one contacts.

-   The SA Plan does not include all "Required Self-assessments" per the
    Exhibit attached to the SBMS Subject Area: Integrated Assessment
    (Issued: January 2003).

> **Recommendation:** The ALD should ensure that the appropriate
> "Required Self-assessments" from the Subject Area Exhibit are reviewed
> and incorporated, as appropriate, into the SA Plan.

Two additional items were noted during interviews with CEGPA managers.
It was indicated by interviewees that these items are under review and
will be addressed. The recommendation proffered here is that CEGPA
proceed as managers indicated in addressing these items:

> CEGPA Decision Papers should be implemented throughout the
> Directorate.
>
> Any duplication of effort between CTAP and DOE's Program Review of OEP
> should be minimized.

### IAP Management System Programmatic Issues

It is acknowledged that some of the issues discussed below are known to
BNL management and the IAP Management System Steward and Point of
Contact, and that efforts may be underway to address these issues. These
issues are noted here for emphasis.

SBMS Subject Area: Integrated Assessment, 2. Annual Review of
Organizational Performance, Required Procedure, Step 1 states, "In
September the Integrated Assessment Program Point of Contact prepares
and distributes a reminder/guidance memorandum and distributes it to
positions identified on the Laboratory Organizational Chart. This
memorandum serves to remind organizations to conduct their annual review
and provides any special circumstances that may be unique to the review
period." In the past, this memorandum has also provided a reminder that
organizations are to complete annual updates of their self-assessment
programs by December 31 as stated specifically in the Subject Area.
Interviewees did not recall receiving this memorandum in September 2002
or subsequently. IO found no evidence that this reminder memorandum was
distributed.

Interviewees noted the lack of timeliness in receiving feedback from DOE
regarding BNL's Year End Self-Evaluation - Fiscal Year 2002, and thus
not being able to respond to said feedback. One similar comment was also
noted regarding the lack of timeliness in DOE's feedback regarding the
draft FY03 Critical Outcomes.

[]{#__RefHeading___Toc38940660 .anchor}Exhibit 1 - Documents Reviewed

1.  CEGPA Directorate Self Assessment Plan for FY2003 (2/24/03)

2.  CTAP 2002 \[Briefing Book\], September 23 -- 24, 2002

3.  \[Draft\] Peer Review Report, Communications and Trust Advisory
    Panel, October 2002

4.  CEGPA Critical Outcome Tracking \[spreadsheet\]

5.  Brookhaven National Laboratory Fiscal Year 2003 Mid-Year Summary
    Status \[presentation at DOE/BSA Partner's Retreat\], April 10, 2003

6.  FY-04 Critical Outcomes and Supporting Objectives \[presentation at
    DOE/BSA Partner's Retreat\], April 10, 2003

7.  Summer Sundays, CEGPA Decision Paper, September 2002

8.  Laboratory Link, CEGPA Decision Paper, November 2002

9.  High School Tours, CEGPA Decision Paper, December 10, 2002

10. Assessment of Outreach to Civic Organizations, CEGPA Decision Paper,
    December 16, 2002

11. Community Involvement Office Web Site

12. Media and Communications Office Web Site

13. Office of Educational Programs Web Site

14. Office of Educational Programs Self-Assessment for FY 2002,
    9/18/2002

15. Lab Self Evaluation to the Office of Science, Workforce Development
    for Teachers and Scientists \[spreadsheet template for DOE FY03
    Review\]

16. Workforce Development for Teachers and Scientists at Brookhaven
    National Laboratory \[presentation at DOE On-Site Review\]

17. U. S. Department of Energy Undergraduate Research Fellowships Summer
    2002 Evaluation Report, Center for Workforce Development

18. U. S. Department of Energy Office of Science Program Guidebook,
    Energy Research Undergraduate Laboratory Fellowship (ERULF) and
    Community College Institute of Science and Technology (CCI)

19. U. S. Department of Energy Office of Science Program Guidebook,
    Pre-Service Teacher Program (PST)

20. Public Affairs/Community Relations, Weekly Report Forecast,
    2/14/03-2/20/03

21. Public Affairs/Community Relations, Weekly Report Forecast,
    2/21/03-2/27/03

22. Brookhaven National Laboratory Community Involvement Plan,
    BNL-52562, April 15, 1999

[]{#__RefHeading___Toc38940662 .anchor}Exhibit 2 -- Personnel
Interviewed

1.  M. Lynch, ALD CEGPA

2.  K. White, Special Assistant to the ALD CEGPA

3.  K. Geiger, Manager, Community Involvement Office

4.  M. Rowe, Manager, Media and Communications Office

5.  B. Murfin, Director of Educational Programs

6.  G. Schroeder, Web Services

[]{#__RefHeading___Toc38940663 .anchor}Exhibit 3 - Documents Referenced

1.  SBMS Management System Description: Integrated Assessment Program,
    Issue Date: July 1999

2.  BNL Independent Oversight Office FY 2003 Program Plan: Review of
    Organizational Self-Assessment Programs (December 12, 2002)

3.  U.S. Department of Energy Contract with Brookhaven Science
    Associates, DE-AC02-98CH10886

4.  Appendix B (to DE-AC02-98CH10886), Critical Outcomes, Objectives,
    and Performance Measures FY 2003 Brookhaven National Laboratory
    (04/18/03)

5.  Year End Self-Evaluation - Brookhaven National Laboratory - Fiscal
    Year 2002

6.  Brookhaven National Laboratory Draft Institutional Plan FY 2003 --
    FY 2007, May 2002

7.  SBMS Subject Area: Integrated Assessment, Effective Date: January
    2003

8.  SBMS Subject Area: Internal Controlled Documents, Effective Date:
    May 2002

9.  SBMS Subject Area: Performance Goals and Measures for Employees,
    Effective Date: October 2001

10. SBMS Subject Area: Performance Appraisals, Effective Date: September
    2002

11. SBMS ESH Standard 1.2.1, Corrective Action Management and Tracking
    for External and Internal Assessments, Rev. 5, Effective Date:
    December 2002

12. Brookhaven National Laboratory Independent Oversight Report SA
    00-06, Evaluation of the Community Involvement, Government, and
    Public Affairs Directorate (September 26, 2000)