**Root Cause Analysis**

**Large Hadron Collider**

**Magnet System Failure**

**Prepared For**

**Fermilab Research Alliance (FRA)**

**September 14, 2007**

[]{#__RefHeading___Toc174459491 .anchor}**Acknowledgements**

The team wishes recognize the efforts of the individuals contributing to
the US LHC project. It is clear to the Root Cause Analysis Team (RCAT)
that issues that arose during the LHC project were not from a lack of
dedication, expertise of the individuals involved in the design and
fabrication, or desire to see the LHC a success. All individuals
interviewed were extremely dismayed at the events that have occurred and
were dedicated to making expeditious and technically-correct repairs.

Fermilab, CERN, LBNL, BNL and DOE professionals encountered by the team
were open and frank during interviews and in written statements, and
seemed genuinely interested in being part of the solution to reduce the
likelihood of recurrence. The team greatly appreciates the support
demonstrated and the Fermilab commitment to the subsequent success of
this root cause analysis.

[]{#__RefHeading___Toc174459492 .anchor}**Signature Page**

**Submitted: Approved:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
[\_\_//Signature on file \_\_\_\_\_\_\_\_\_\_\_\_\_]{.underline}

Jeffrey Cotton, EG&G Louis Smith, EG&G

Program Manager Director, Performance Management

[\_\_//Signature on file \_\_\_\_\_\_\_\_\_\_\_\_\_]{.underline}
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Jim Eckhardt, URS John Robert Grant, Fermilab

Quality Assurance Manager Head, Office of Quality and Best

Practices

[\_\_//Signature on file \_\_\_\_\_\_\_\_\_\_\_\_\_]{.underline}
**Acknowledged:**

Michael Hirsh, URS

Vice President; Program Manager

[\_\_//Signature on file \_\_\_\_\_\_\_\_\_\_\_\_\_]{.underline}
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Gordon Kennedy, EG&G Piermaria Oddone, Fermilab

Director, Mission Assurance Director, Fermilab

[\_\_//Signature on file \_\_\_\_\_\_\_\_\_\_\_\_\_]{.underline}
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

John Mammosser, SNS Stephen Holmes, Fermilab

Superconducting Cavity Technical Associate Director, Accelerator

Engineer Division

[\_\_//Signature on file \_\_\_\_\_\_\_\_\_\_\_\_\_]{.underline}

Lou Tucciarone, URS

Sr. Vice President; Regional Manager

[**Acknowledgements** 2](#__RefHeading___Toc174459491)

[**Signature Page** 3](#__RefHeading___Toc174459492)

[**Executive Summary** 6](#__RefHeading___Toc174459493)

[***Event*** 6](#__RefHeading___Toc174459494)

[***Direct and Root Causes*** 6](#__RefHeading___Toc174459495)

[***Contributing Causes*** 6](#__RefHeading___Toc174459496)

[***Root Cause Impact*** 6](#__RefHeading___Toc174459497)

[***The Case for Change*** 7](#__RefHeading___Toc174459498)

[***Recommendations*** 8](#__RefHeading___Toc174459499)

[***Conclusion*** 8](#__RefHeading___Toc174459500)

[***Recommended Actions*** 9](#__RefHeading___Toc174459501)

[**Description of Events** 11](#__RefHeading___Toc174459502)

[**Purpose and Scope** 11](#__RefHeading___Toc174459503)

[**Approach** 11](#__RefHeading___Toc174459504)

[**Report Arrangement** 12](#__RefHeading___Toc174459505)

[***Root Cause Process*** 13](#__RefHeading___Toc174459506)

[**Quadrupole Cause Summary** 14](#__RefHeading___Toc174459507)

[**Quadrupole Cause Summary Discussion**
15](#__RefHeading___Toc174459508)

[***Direct Cause*** 15](#__RefHeading___Toc174459509)

[1)a.i.1.a - Equipment Environment Not Considered
15](#__RefHeading___Toc174459510)

[***Contributing Causes*** 15](#__RefHeading___Toc174459511)

[1)a.i.2 - Specifications Less Than Adequate
15](#__RefHeading___Toc174459512)

[1)a.ii.1 - Independent Review Less Than Adequate
15](#__RefHeading___Toc174459513)

[2)a.i.1 - No Procedure 16](#__RefHeading___Toc174459514)

[2)a.i.2 - Procedure Use Not Required But Should Be
17](#__RefHeading___Toc174459515)

[2)b.i.1 - Task Not Analyzed 18](#__RefHeading___Toc174459516)

[2)c.i.1 - Inspection Instructions Less Than Adequate
19](#__RefHeading___Toc174459517)

[2)c.i.2 - Inspection Techniques Less Than Adequate
19](#__RefHeading___Toc174459518)

[2)d.i.1 - No Method Available -- Not Required
20](#__RefHeading___Toc174459519)

[2)e.ii.1 - Audits And Evaluations (A & E) Lack Depth
20](#__RefHeading___Toc174459520)

[**Heat Exchanger Cause Summary** 21](#__RefHeading___Toc174459521)

[**Heat Exchanger Cause Summary Discussion**
21](#__RefHeading___Toc174459522)

[***Direct Cause*** 21](#__RefHeading___Toc174459523)

[3 a.i.1.a - Equipment Environment Not Considered
21](#__RefHeading___Toc174459524)

[***Contributing Causes*** 22](#__RefHeading___Toc174459525)

[3)a.ii.1. Independent Review Less Than Adequate
22](#__RefHeading___Toc174459526)

[3)b.i. Quality Control 22](#__RefHeading___Toc174459527)

[**Common Cause Summary Discussion** 23](#__RefHeading___Toc174459528)

[***Contributing Causes*** 23](#__RefHeading___Toc174459529)

[2)e.i.1 - Standards, Policies, or Administrative Controls Incomplete
23](#__RefHeading___Toc174459530)

[**Root Cause Discussion** 27](#__RefHeading___Toc174459531)

[2)e.i.1 - Standards, Policies, or Administrative Controls Incomplete
27](#__RefHeading___Toc174459532)

[**Recommendations** 28](#__RefHeading___Toc174459533)

[***Project Planning*** 29](#__RefHeading___Toc174459534)

[***Design and Engineering Process*** 31](#__RefHeading___Toc174459535)

[*Design Reviews:* 32](#__RefHeading___Toc174459536)

[*Design Process Management* 32](#__RefHeading___Toc174459537)

[***Training*** 33](#__RefHeading___Toc174459538)

[***QA Program*** 33](#__RefHeading___Toc174459539)

[**Supporting Documentation** 34](#__RefHeading___Toc174459540)

[**Attachment 1 -- Root Cause Analysis Flowchart**
36](#__RefHeading___Toc174459541)

[**Attachment 2 -- Personnel Interview List**
38](#__RefHeading___Toc174459542)

[]{#__RefHeading___Toc174459493 .anchor}**[Executive
Summary]{.underline}**

[]{#__RefHeading___Toc174459494 .anchor}***Event***

Fermilab led an inter-laboratory project to provide critical components
for the Large Hadron Collider (LHC) at the CERN Laboratory in Geneva,
Switzerland. During the initial hardware commissioning phase there were
two failures of the Fermilab supplied equipment.

A root cause analysis (RCA) was performed to determine the cause of the
failures and to offer recommendations to reduce the risk of recurrence
during future design and manufacturing initiatives.

[]{#__RefHeading___Toc174459495 .anchor}***Direct and Root Causes***

The RCA Team (RCAT) determined common direct and root causes are
applicable to both failures:

*Direct Cause* -- In both instances the Fermilab design process did not
account for the environment in which the assembled components would
operate. In the first event an improperly designed heat exchanger pipe
failed during a test at less than the 20 bar static normal operating
limit. In the second event the application of design maximum pressure of
25 bar (asymmetric) pressure caused longitudinal movement of quadrupole
magnets and subsequent breakage of internal support structures and
cryogenic connections. The failure occurred at less than 20 bar because
the dynamic application of pressure was not considered in the design of
the support structure.

*Root Cause* -- Fermilab engineering management controls do not include
codified, standard design process requirements that include an
integrated systems design, design review, and documentation recording
and archival process. Instead, Fermilab relies upon individual
contributors to obtain review of design basis calculations and
recognition of interface and integration requirements. In both
instances, the lack of documentation and in-depth review resulted in
critical design errors being missed until the components were tested *in
situ* at CERN.

[]{#__RefHeading___Toc174459496 .anchor}***Contributing Causes***

There are a number of contributing causes that can be described as
failed or non-existent barriers. These barriers, if they had been
established or used effectively, should have caught the design errors
prior to the systems tests at CERN. These are addressed as contributing
causes in the body of the report.

[]{#__RefHeading___Toc174459497 .anchor}***Root Cause Impact***

Putting this project in perspective, the published overrun of the
project, including the repairs to the Fermilab components, is less than
10% of the total project cost. Although costly in terms of dollars
spent, this overrun is not extraordinary and is within industry averages
for projects of this size and complexity. Even so, Fermilab commissioned
a root cause analysis. Clearly, the concern is not simply project cost
or schedule; rather it is the perception of Fermilab's technical
engineering capabilities and its ability to manage large, complex
projects. The inference being that if Fermilab cannot manage the
technical quality of one portion of the LHC program without discovering
and eradicating design errors prior to fabrication, how can it be
entrusted to manage other large projects such as Project 'X' and the
International Linear Collider?

[]{#__RefHeading___Toc174459498 .anchor}***The Case for Change***

Fermilab is unique in the high energy physics (HEP) community in that it
has, for years, been the operator of the most powerful particle
accelerator in the world. It now finds itself on the cusp of a time when
that honor is going to pass to another facility on another continent. As
such, several key decisions are being made regarding Fermilab: whether
to shut down or modify the Tevatron for other use, whether to fund new
experiments, and the level of involvement in international
collaborations such as Project 'X' and the ILC.

Further complicating matters, Fermilab's most senior, knowledgeable
personnel are of, or are approaching, retirement age. Unfortunately, a
knowledge database or engineering lessons learned program has never been
created to pass on the information and basis for decisions and design to
incoming physicists and engineers. This detracts from one of the key
discriminators that Fermilab has today -- being one of the most
experienced designers and fabricators of large machines in the high
energy physics community.

In order to remain a viable laboratory Fermilab must have the reputation
of delivering positive scientific results while exhibiting good
business, engineering, and project management practices.

These concerns dictate the need for a cultural shift. In order to
maximize the availability of funds to support R&D and other new
programs, and to make the most use of available funds, Fermilab must
improve its processes. This does not mean limiting the freedom of
scientific personnel to do research and development; it does mean that
there needs to be a shift in perspective: capture and archive decisions
and results of testing for future users, engineers, and scientists;
changing the conduct of engineering reviews to focus on design basis and
component integration; improving project management practices such as
being more aware of risk and how to mitigate it; and managing for
successful outcomes the first time thereby limiting the amount of
re-work and expense.

In summary, it is incumbent on Fermilab to ensure its engineering and
management processes and controls are among the best in the HEP
community; well established and well-executed. By demonstrating such
technical and management excellence, Fermilab can improve its chances of
remaining a viable laboratory well into the future.

[]{#__RefHeading___Toc174459499 .anchor}***Recommendations***

Recommendations are discussed in the body of the report and include
suggestions for improvement in five areas:

-   Project planning and execution

-   Design and engineering processes

-   Training

-   Quality Assurance

-   Need to establish inter-laboratory and international project
    collaboration processes

Generally, the recommendations can be implemented with minimal overall
cost and/or in lieu of current requirements or processes thereby
limiting the addition of new layers of administration. For brevity, a
compiled list of recommended actions without discussion is presented on
the next page.

[]{#__RefHeading___Toc174459500 .anchor}***Conclusion***

The Team appreciates the contribution of the LHC project participants
during the conduct of the RCA. The participants were open and responsive
to questions and provided constructive feedback when recommendations
were discussed. The Team believes that the recommendations, if adopted,
will complement the impressive skills, experience and motivation of the
Fermilab professionals and position them for future projects and
laboratory viability.

[]{#__RefHeading___Toc174459501 .anchor}***Recommended Actions***

-   Project planning and execution

    -   Enhance the project kickoff process to include:

        -   Approach

        -   Risk identification and mitigation

        -   Project constraints (budget, schedule, manpower, and
            operational issues)

        -   Key project members roles and responsibilities

        -   Identification of inter-divisional participation

        -   Test plans and testing environment

        -   Identification of known operational environment and
            constraints

        -   Clear identification of specifications or design
            requirements

    -   Enhance Project Controls to include:

        -   EVMS used for project management

        -   Development of more comprehensive Basis of Estimates for all
            portions of the project and ensure that engineering and
            fabrication quality assurance is included adequately in the
            estimates.

            -   Develop and maintain a resource-loaded baseline schedule

        -   Risk-based change management process

        -   Refined contingency development and management processes to
            include use of management reserves

        -   Require that all project schedules include specific
            activities, milestones and adequate time allowances for:

            -   Engineering review, testing and other QA activities

            -   Formal break point milestone to transition from R&D to
                production

            -   Testing and test program development

        -   Establish formal project documentation usage requirements

            -   Re-visit risk analysis at prescribed intervals to ensure
                project issues are known and resolved

    -   Establish risk management position for participation in all
        projects of significant size or complexity

-   Design and Engineering Process

    -   Establish a Fermilab Chief Engineer position reporting to the
        Director

    -   Require that all project organizations include a specific role
        for systems integration manager

        -   Define integration requirements in engineering control
            documentation

    -   Create and manage a laboratory wide Fermilab design/engineering
        manual and implementing procedures including configuration
        control systems and processes

        -   Design reviews

            -   Establish templates and checklists that must be included
                in design reviews

            -   Perform reviews aimed at determining technical
                specifications are met

        -   Establish requirements to provide independent technical
            reviews

        -   Establish design review formats and designate the
            appropriate time to allow checks of design calculations

        -   Define archival requirements for design basis calculations
            and R&D results

        -   Define required internal design basis calculation
            cross-checks and review

    -   Develop a risk-based change management process for use during
        design efforts

    -   Differentiate critical design elements and how to control them

-   Training

    -   Establish or procure formal training for Project managers

    -   Develop orientation for personnel involved with design efforts
        to familiarize them with Fermilab processes and procedures.

-   Quality Assurance

    -   Establish a risk-based QA program at Fermilab (underway)

    -   Establish a lessons learned program and database to capture and
        share lessons from previous projects among all collaborating
        labs

    -   Establish key QA milestones in project budgets and schedules

    -   Establish a QA inspection program that includes minimum
        standards for inspection of manufactured articles and
        identification of quality assurance inspectors

-   Commission a Process Improvement Team to recommend project execution
    guidelines and requirements for Fermilab project managers to follow
    for both inter-laboratory and international collaborative projects.

    []{#__RefHeading___Toc174459502 .anchor}**[Description of
    Events]{.underline}**

Fermilab led a consortium of three U.S. laboratories (Fermilab,
Brookhaven National Laboratory, and Lawrence Berkeley National
Laboratory) that have supplied critical components to the Large Hadron
Collider, currently entering its commissioning phase at the CERN
Laboratory in Geneva, Switzerland. These components were designed,
fabricated, and delivered to CERN over the period 1998-2006. During the
initial hardware commissioning phase there were two failures of Fermilab
supplied equipment:

1)  On November 25, 2006 a heat exchanger internal to one of the
    Fermilab supplied magnets collapsed in a pressure test

2)  On March 27, 2007 structural supports internal to one of the
    Fermilab supplied magnets failed in a pressure test.

    []{#__RefHeading___Toc174459503 .anchor}**[Purpose and
    Scope]{.underline}**

The Fermilab Director commissioned a root cause analysis (RCA) to
determine the cause(s) of the failure and to offer recommendations to
prevent similar issues from recurring during future design and
manufacturing initiatives. More specifically the RCA team was charged to
undertake an investigation and analysis in the following areas:

-   Conduct a root cause analysis that identifies how deficiencies in
    the quadrupole support structures, and the heat exchanger, went
    unrecognized until the components were under final test at CERN.

-   Recommend changes to Fermilab's procedures and processes to prevent
    recurrence.

    []{#__RefHeading___Toc174459504 .anchor}**[Approach]{.underline}**

The RCA team consisted of independent experts from Fermilab, Spallation
Neutron Source (SNS), and EG&G/URS to perform the RCA as shown in
Attachment 1. None of the team members had been directly involved in the
LHC magnet project. The team reviewed project documentation, Fermilab
procedures manuals, and conducted interviews of Fermilab, Berkeley,
Brookhaven, and CERN project personnel and Fermilab support staff, in
eight specific areas:

-   Project Management

-   Agreements

-   Specifications

-   Design

-   Procurement & Construction

-   Acceptance & Testing

-   Delivery

-   Commissioning & Startup

A list of personnel interviewed as part of this root cause analysis is
contained in Attachment 2.

The team, as charged, reviewed design, fabrication, assembly, QA and
testing processes used to produce the Fermilab LHC Interaction Region
Magnets and integrated DFBXs to determine the causes of failures and to
make recommendations to reduce the likelihood of recurrence.

Causes were identified using a variety of root cause tools including
TapRoot®[^1], five-why's and Ishikawa[^2] diagramming. As part of the
root cause, the team performed a review to answer three basic questions
about Fermilab project execution practices:

1)  Did Fermilab follow its own processes?

2)  Do Fermilab processes follow best practices?

3)  What changes are recommended to align Fermilab processes with
    generally accepted standards and best practices?

Finally, the team made every attempt to make sure that recommendations
are targeted and actionable. The recommendations are intended to revise
or enhance current requirements; thereby reducing the cost of
implementation to make them more practicable for Fermilab.

[]{#__RefHeading___Toc174459505 .anchor}**[Report
Arrangement]{.underline}**

The report presents the causes for the Heat Exchanger and Quadrupole
failures. The direct and contributing causes are related, but are
presented separately for clarity.

For purposes of this report, it is understood that the quadrupole magnet
did not fail; rather the cold mass support structure elements were
damaged when the cold mass moved under pressurization test. For ease of
understanding, the report uses nomenclature commonly used at Fermilab;
calling it the quadrupole failure.

The document contains sections for each, arranged as follows:

*Quadrupole Cause Summary* -- Presents summary level causes (direct,
contributing, and root) in outline form, as identified using the
TapRoot® process.

*Quadrupole Cause Summary Discussion* -- Following the Cause Summary
outline, the discussions are arranged into categories of Direct and
Contributing Causes.

*Heat Exchanger Cause Summary* -- Presents summary level causes (direct,
contributing, and root) in outline form, as identified using the
TapRoot® process.

*Heat Exchanger Cause Summary Discussion* -- Following the Cause Summary
outline, the discussions are arranged into categories of Direct and
Contributing Causes.

*Common Cause Summary* -- Contributing Causes germane to both incidents

*Root Cause Discussion* -- Presentation of root cause of the failures.

*Recommendations* -- Solutions to problems discussed

*Supporting Documentation* -- Explanation of documentation used as a
source of information when assessing performance and to identify Direct,
Contributing, and Root Causes.

[]{#__RefHeading___Toc174459506 .anchor}*Root Cause Process*

In discussions with Fermilab management prior to starting this analysis,
it was agreed that the team would (1) use the TapRoot® system, a very
structured process to identify causes of an event or issue and (2) focus
on the adequacy of Fermilab's processes. TapRoot® was chosen because it
uses fault tree analysis to determine the cause of an issue that drives
participants beyond the human performance to the cause behind the
performance, expressed in terms of system or process issues.

The TapRoot® system guides reviewers performing root cause analysis
through fault trees in both equipment design and human performance
factors. The TapRoot® analysis related to human factors considers
individual, team, and management performance. The team used these
factors to develop common causal factors related to system and process
issues as depicted in the quadrupole and heat exchanger outlines.

Direct and Contributing Causes are presented separately for the
Quadrupole and Heat Exchanger failures.

[]{#__RefHeading___Toc174459507 .anchor}**[Quadrupole Cause
Summary]{.underline}**

The TapRoot® standard terms are presented in the outline below. To be
consistent with the TapRoot process, the team is using these standard
terms, though in the discussion following the outline we have used the
TapRoot standard terms in context of Fermilab processes

1)  Equipment Difficulty

    a.  Design

        i.  Design Specifications

            1.  Problem Not Anticipated

                a.  Equipment Environment Not Considered

            2.  Specifications Less Than Adequate

        ii. Design Review

            1.  Independent Review Less Than Adequate

2)  Human Performance Difficulty

    a.  Procedures

        i.  Not Used / Not Followed

            1.  No Procedure

            2.  Procedure Use Not Required But Should Be

    b.  Training

        i.  No Training

            1.  Task Not Analyzed

    c.  Quality Control

        i.  Quality Control Less Than Adequate

            1.  Inspection Instructions Less Than Adequate

            2.  Inspection Techniques Less Than Adequate

    d.  Communications

        i.  No Communication or Not Timely

            1.  No Method Available -- Not Required

    e.  Management System

        i.  Standards, Policies, or Administrative Controls

            1.  Incomplete

        ii. Oversight / Employee Relations

            1.  Audits And Evaluations (A & E) Lack Depth

            2.  A & E Not Independent

[]{#__RefHeading___Toc174459508 .anchor}**[Quadrupole Cause Summary
Discussion]{.underline}**

This section discusses the causal factors used in the TapRoot® system in
context of Fermilab systems and processes. Each of the headers used in
this section tie directly to the causal factor outline.

[]{#__RefHeading___Toc174459509 .anchor}***Direct Cause***

[]{#__RefHeading___Toc174459510 .anchor}[1)a.i.1.a - Equipment
Environment Not Considered]{.underline}

The Fermilab LHC Interaction Region (IR) magnets were designed as a
collection of components that were assembled as a "quadrupole magnet"
system. The design effort did not consider the components together as an
assembled system (i.e. triplet with an integral DFBX) and did not
analyze the assembled component system in its intended, operational
environment. The result was the pressure vessels and support structures
were built to withstand the static pressure requirements, but did not
consider the application of the pressure under dynamic load. Under test
in the LHC tunnel, the quadrupole magnets moved longitudinally damaging
internal support structures.

[]{#__RefHeading___Toc174459511 .anchor}***Contributing Causes***

[]{#__RefHeading___Toc174459512 .anchor}[1)a.i.2 - Specifications Less
Than Adequate]{.underline}

Early in the project, CERN provided boundary conditions and known
specifications; however, other required specifications were not defined
until much later in the design and construction phases. As would be
expected during a research and development effort, detailed
specification development is part of the ongoing process; however,
issues evolved that caused concern with the engineers fulfilling the
design and fabrication role:

1.  Identification of interface and operational/performance
    specifications did not characteristically precede the development
    and fabrication of equipment.

    a.  Although specifications were developed early in the production
        cycle as drafts, they evolved during the production phase and
        were not finalized until late in the project cycle.

    b.  Engineering staff focused on the specifications for major
        components such as magnets in a linear fashion, and not
        adequately in advance of the production effort to allow
        independent reviews and approvals.

        []{#__RefHeading___Toc174459513 .anchor}[1)a.ii.1 - Independent
        Review Less Than Adequate]{.underline}

The independent review process used at Fermilab varied by component and
technical specialty, was not well documented, and was not comprehensive.

Engineers are not required to have their design basis calculations
reviewed during the R&D process. Any internal design review that did
occur was based upon the judgment of the individual engineers and
scientists, and not specified by Fermilab design process procedures, the
project QA plan, or team leaders. The reviews were not comprehensive
with respect systems analysis, inter-disciplinary participation and
engagement of technical specialists who were not affiliated with the
respective national labs and universities participating in the LHC
Program.

The programmed independent reviews (conceptual design review, two
engineering design reviews, and production readiness review) were
scheduled around critical components and not system assemblies. The
reviews were typically PowerPoint® presentations that focused on issues
or challenges that the design component leader selected. The reviews did
not validate assumptions, evidence that specifications were being
adequately met, or verify design basis calculations and/or conduct a
detailed review of drawings. Although issues were captured and
corrective actions taken, the reviews were not performed at a level of
detail sufficient to allow the capture of the errors in the Fermilab
supplied components. While a PowerPoint® type of review is adequate for
conceptual design, it does not, by virtue of its presentation format,
accommodate a comprehensive review of complex components and systems as
required during engineering design or readiness reviews.

Design reviews and design review presentations were performed by
personnel recognized as experts within the high energy physics
community. When presenters are recognized experts, the reviewers' level
of in-depth questioning was less probing in deference to the
individual's stature. Furthermore, while some of these experts may have
been independent from the perspective of not having worked personally on
the LHC project, they knew the project participants well. In many cases
they would have their projects reviewed by the same Fermilab personnel
being reviewed. This created a "reciprocal" situation of mutual benefit
and removed an element of independence, as in a professional courtesy
afforded personnel that work in the same industry.

Personnel interviewed pointed out that they had initiated an in-depth
design review of the Berkeley DFBX design when errors were discovered
but failed to conclude that such a review would be appropriate for the
design work being accomplished at Fermilab.

The review process is characterized by the following three issues:

1.  There is no requirement that drives internal review of design basis
    calculations.

2.  There is no comprehensive process to drive an integrated,
    system-focused, internal project review during design/development.

3.  The external design reviews are neither detailed nor independent.

    []{#__RefHeading___Toc174459514 .anchor}[2)a.i.1 - No
    Procedure]{.underline}

The engineering design process is not codified or standardized at
Fermilab resulting in a wide variety of approaches being used by
designers of the various components.

Sub-elements of the design process are codified in the Fermilab
Environmental, Safety, and Health Manual (FESHM). The FESHM requires the
capture of summary-level information for designs that affect life-safety
code, such as pressure vessels, in engineering design notes.
Nevertheless components that require FESHM compliance do not require
archival of standardized, organized and documented design assumptions,
design calculations, or ideas and concepts tried and abandoned.

Documentation of design efforts vary by individual. Some keep exhaustive
notes and ledgers containing basis of design while others do not.
Further there is no central project record with design development
documentation for future use or reference.

This decentralized component- and discipline-based design approach
affected the systems integration with Q1s and Q3s coupled with Q2s.
Interviews disclosed that the stress calculations for the Q2s were
assumed to be representative and therefore applicable for Q1s and Q3s.
This was incorrect because the pressure loads/stress was symmetrical for
the Q2s and asymmetrical for the Q1s and Q3s when coupled with Q2s.
Because the design approach focused on components and component systems
there was no opportunity to validate or invalidate this design
assumption based on a Systems Integration assessment.

Final design drawings are controlled and archived, but the engineering
basis is not captured for use by others or outside the responsible
organizational division.

[]{#__RefHeading___Toc174459515 .anchor}[2)a.i.2 - Procedure Use Not
Required But Should Be]{.underline}

Technical Division (TD) has a Quality Assurance manual and more
specifically the US LHC had a Quality Assurance manual, TD-2010, Version
2, that contains requirements for design phase tasks including
documentation, reviews and archival of design basis information, and
interface requirements. For the LHC, the Quality Assurance manual
procedures were followed for the fabrication and assembly phase (as the
manual is largely a codification of the "Traveler" process). The
procedures presented in the manual were only partially (or not at all)
applied during the research and development or design phase activities.
The interviewees that knew of the document's existence confirmed this
process, stating that the QA Manual is widely accepted as pertaining
only to fabrication.

As the manual provides latitude for the PM to apply the requirements of
the document to the project in the preamble, no violation of Fermilab
procedure or policy occurred; however, conformity with the QA manual
during the design phase, especially in the areas of cross-checking
design basis calculations and integration, may have prevented the
occurrence of the quadrupole failure.

Aside from the use and conformity with the processes described in the TD
QA manual, the manual in and of itself was not structured for effective
QA. As told to the team, the manual was developed to both codify TD QA
practices and to cross reference Fermilab QA practices with CERN QA
requirements. The QA manual was not developed to define QA processes
customized and relevant to the multi-lab, multi-component and
multi-national aspects of the US LHC project.

[]{#__RefHeading___Toc174459516 .anchor}2)b.i.1 - Task Not Analyzed

"Task Not Analyzed" is a title from the TapRoot® fault tree and
references determination of the training requirements for certain jobs
and their associated tasks. None of the Fermilab tasks discussed in this
section were analyzed for training requirements.

Fermilab's on-site training programs are largely focused on Safety,
Orientation, and Supervisor Training, such as General Employee Training
and General Employee Radiological Training, Confined Space Entry,
supervisory administrative and safety functions. These are used to
ensure personnel are safe to move about the site and to allow entry into
areas of risk. The training programs and new employee orientation do not
guide technical staff in procedures and processes to document design
efforts consistently or to understand the applicability and use of
standards such as FESHM. In addition, Fermilab does not provide project
managers with training on tools and techniques for guiding projects from
idea and concept to successful completion such as integrated research
and development efforts, team based reviews, and project controls
(scheduling, organization, cost and budget management and information
systems management).

*Project Management* -- Project management is not a professional
progression employment path at Fermilab, nor is it a position that
personnel seek to occupy. Instead, the personnel that have championed a
new concept or new experiment being considered are typically asked to
serve as the project manager.

In general, personnel performing project management functions do so as a
collateral duty while maintaining their primary focus on their areas of
technical expertise. The newly established Fermilab Office of Project
Management Oversight provides directional guidance to project managers
in the earliest phases of a new experiment, or project, but does not
provide the resources or functional involvement to guide the day-to-day
operations. Testimony from interviewees indicated that the OPMO reviews
are used to pressure project participants to conform to the Critical
Decision point documentation and informational requirements of Fermilab
and DOE, i.e. DOE O 413.3, Program and Project Management for the
Acquisition of Capital Assets, commonly referred to as "Lehman Reviews."

This approach results in managers performing project management
functions to maintain compliance with reporting requirements at
prescribed intervals rather than using project management tools to more
effectively execute the project. As described in other areas of this
report, project management plans should include tools, such as specific
guidance for project integration and risk management, to help the PM
manage the project. Without proper training in methods such as these,
the PM is not fully equipped.

*Engineering Function* -- The training and orientation process for
technical professionals focuses on administrative HR and Safety issues.
The process does not familiarize technical staff with procedures and
standards to be followed for design, documentation reviews or QA.
Furthermore the procedures and approaches used within TD and within
other Divisions at Fermilab differ. This results in technical design
procedures that are inconsistent within the Lab and varying levels of
granularity in design basis documentation.

*Quality Assurance Inspectors (QAI)* -- There is no formal training and
no formal procedures to standardize the approach to inspect procured and
manufactured items.

Without a standardized approach, there is no consistency in how items
are inspected and tested. With standardization comes process and process
training; ensuring those that are designated to perform this function do
so to consistent levels established by Fermilab.

By setting a qualification standard, personnel will have demonstrated
their proficiency before being relied upon to independently perform
inspections of material received from vendors and to review and observe
testing performed by fabricators such as dye-penetrant testing,
ultrasonic testing, small-gauge measurements, etc.

[]{#__RefHeading___Toc174459517 .anchor}2)c.i.1 - Inspection
Instructions Less Than Adequate

The Traveler document used by Technical Division is an excellent
practice that should be emulated/benchmarked by other laboratories. Even
though it is a best practice, its use of the traveler needs improvement.
The reviews performed by Fermilab design engineers are adequate;
however, inspections by qualified QA personnel should precede the
engineer's signatures to provide documented evidence that the article is
built as required.

[]{#__RefHeading___Toc174459518 .anchor}2)c.i.2 - Inspection Techniques
Less Than Adequate

The Technical Division's use of traveler documents has matured and
provides many positive results for Fermilab as a reference document for
accelerator maintenance and operational enhancements. The travelers use
stop and hold points for sign off by the originating design engineer to
ensure the manufactured item is properly configured before allowing
additional work to continue. The traveler document should however,
contain steps or refer to an approved QA procedure for verification of
fabrication elements. The personnel performing the task should be
qualified in accordance with an approved Fermilab training program,
developed by a Quality Assurance Inspector Level III. This will ensure
consistent inspections by personnel trained to a minimum standard level,
as demonstrated by a documented On-The-Job training program. These
inspections are what the designer should be signing as complete. This
will standardize the review process and free designers to perform other
tasks.

Technical Division has an inspection group as part of the material
control section; however, none are formally trained, qualified, or
periodically re-qualified in accordance with a standard or quality
assurance inspection program. Consequently, inspection efforts focus on
dimensional and documentation checks or procedures in place or as
requested by others: hence inspections are not per a QA program nor
tailored to the project characteristics and requirements.

[]{#__RefHeading___Toc174459519 .anchor}2)d.i.1 - No Method Available --
Not Required

Quadrupole research and design development focused on components and a
sequence of working from the inside out. Design disciplines in the
cryogenics, mechanical, electrical, and physics areas focused on
discrete elements. There were no formal design team meetings where
development and progress issues were discussed in the presence of all
team members and project leaders. Absent these internal group
discussions there were no formal or informal processes to establish
integration between the disparate groups during the development of
individual components that collectively addressed systems integration
issues.

During an R&D effort changes are expected and group discussion is
required to analyze the changes and recognize the effect that one change
has on the upstream and downstream components. Additionally, such
discussions permit the identification of the impact of changes on
previously designed components. Personnel interviewed indicated there
were multiple meetings to discuss single component design issues, but
none of a group manner that looked at the collection of components to
discuss operational and performance design issues.

[]{#__RefHeading___Toc174459520 .anchor}2)e.ii.1 - Audits And
Evaluations (A & E) Lack Depth

Audits and evaluations by project management are designed to make
certain that the project meets minimum requirements of DOE reviews,
neither of which are performed at a level sufficient to discover the
issues related to the LHC Quadruples.

In the case of the LHC project, Fermilab solicited a review by an expert
at Argonne National Laboratory to perform what is now referred to as a
Director's Review. This review addressed issues related to ensuring that
the project was ready for the DOE review, which tends to focus on
program management vice technical issues.

This review resulted in a comment that there was not enough contingency
built into the project and that the project, the way it was constructed,
would not be able to provide the equipment required for the budget
allocated. Immediately following this Fermilab review, the DOE review
concluded the opposite and informed Fermilab that it needed to decrease
the contingency and add scope as it was "over-funded."

Circumstances related to the international nature of the LHC and the DOE
agreement with CERN may have forced the decision on the Fermilab team.
However, no information was presented to demonstrate that the DOE
conclusions were refuted at the time and the resultant action was to
accept the DOE's comment and add scope to the project. This action
contributed to a project overrun reported at \$1.5M of the \$44M
Fermilab portion of the project. Based upon review and interviews, it
was determined that actual costs were much higher as it was Fermilab's
practice to augment projects with personnel charging to general
operating funds. An accounting review revealed that no mechanism exists
to differentiate these costs; and therefore, no way to validate actual
project costs.

LHC Project progress was measured using Earned Values among other
measures. However, design efforts were not budgeted by task and
determining percent complete was often based on time spent or
non-measurable estimates. For example, design personnel reported time
spent without regard to actual work completed and management portioned
the cost over schedule items based upon personal estimation of work
performed. This resulted in the perception of actual performance
measures, but was not indicative of actual project performance.

[]{#__RefHeading___Toc174459521 .anchor}**[Heat Exchanger Cause
Summary]{.underline}**

The TapRoot® standard terms are presented in the outline below. To be
consistent with the TapRoot process, the team is using these standard
terms, though in the discussion following the outline we have used the
TapRoot standard terms in context of Fermilab processes

3)  Equipment Difficulty

    a.  Design

        i.  Design Specifications

            1.  Problem Not Anticipated

                a.  Equipment Environment Not Considered

        ii. Design Review

            1.  Independent Review Less Than Adequate

    b.  Equipment / Parts Defective

        i.  Quality Control

            []{#__RefHeading___Toc174459522 .anchor}**[Heat Exchanger
            Cause Summary Discussion]{.underline}**

            []{#__RefHeading___Toc174459523 .anchor}***Direct Cause***

            []{#__RefHeading___Toc174459524 .anchor}3 a.i.1.a -
            Equipment Environment Not Considered

The LHC Interaction Region heat exchangers were not designed to handle
the 20 bar pressure to which they are subjected to during cool-downs and
cryogenic system operational conditions.

The prototype design for this heat exchanger was a variant of a heat
exchanger design used at CERN, which consisted of corrugated copper
tubing brazed to a stainless steel end flange. A prototype, consisting
of a short section of this tubing and mating flange, was pressure tested
at FNAL to determine if the material was adequate for this design.
During pressure testing of this short section, the tubing started to
fail near the braze. The brazing annealed the first few convolutions of
tube softening the material near the ends. In response to this observed
failure, the design was modified by adding a tube to reinforce the ends.
This revised design resulted in a successful pressure test*.*

Between the time when heat exchanger testing ended and the components
were fabricated another engineer was assigned. He was given the
responsibility to complete the design and procure the Heat Exchanger
components. The revised dimension of the stainless steel sleeve was not
noted in the Engineering Note and Drawings. Procurement and fabrication
drawings did not indicate revised dimension of the sleeve support for
the annealed section of copper. With design development and reviews not
following a formal process, this change went unnoticed.

There was no pressure test of the fabricated, assembled heat exchanger
unit prior to delivery to CERN and the error wasn't discovered until
CERN had installed the Fermilab components into the LHC. During a
pressure test, the units started to fail. Fermilab did pressure test the
pressure vessel, which contains the heat exchanger, but the heat
exchanger was open resulting in the same pressure being applied on both
the inside and outside. As explained by interviewees, the test did not
involve a differential pressure between the inside and outside of the
heat exchanger as the heat exchanger itself was not considered a
pressure vessel.

[]{#__RefHeading___Toc174459525 .anchor}***Contributing Causes***

[]{#__RefHeading___Toc174459526 .anchor}3)a.ii.1. Independent Review
Less Than Adequate

The intended modification was not carried through to the final design.
The original design was used as the specification to the vendor, rather
than the modified design that resulted from the pressure test failures.
In addition, the engineer did not specify a means of controlling the
distance or the amount of the annealing during the fabrication.

The result is that the copper tubing was annealed beyond where the two
inch stainless end flange supported it. During a pressure test at CERN,
the heat exchangers started to fail, and upon inspection of the untested
heat exchangers, several showed signs of damage. The damage to these
units is believed to have happened during the cryogenic testing of the
magnets at Fermilab.

Upon review of the final design drawings, it was determined that the
heat exchanger components were designed and reviewed by the same
engineer. There was no independent design review of the heat exchanger
mechanical design and therefore no opportunity for another person
familiar with the behavior of materials in a cryogenic, pressure-filled
atmosphere to check the work done and highlight issues.

[]{#__RefHeading___Toc174459527 .anchor}3)b.i. Quality Control

The first article testing of the heat exchangers produced in industry
did not include vendor or in-house pressure testing requirements nor was
there technical representation from Fermilab at the vendor site during
the heat exchangers production runs. Testing should have discovered the
design error and technical representation may have discovered the
annealing or support issue, if it was present during fabrication.

Additionally, the prototype heat exchangers were tested to 7 bar at
Fermilab, not the 20 bar operational pressure found at CERN. The tests
at Fermilab were focused on several aspects other than pressure in
operational conditions at CERN. Fermilab should have required testing of
the production run and been at the manufacturer during fabrication;
however, it was assumed that the fabrication would be adequately handled
by the vendor.

The issue was further compounded because CERN's tests of the heat
exchangers were focused on their ability to handle the thermal loads and
also failed to perform pressure testing prior to installation in the
LHC.

[]{#__RefHeading___Toc174459528 .anchor}**[Common Cause Summary
Discussion]{.underline}**

[]{#__RefHeading___Toc174459529 .anchor}***Contributing Causes***

[]{#__RefHeading___Toc174459530 .anchor}2)e.i.1 - Standards, Policies,
or Administrative Controls Incomplete

Fermilab does not provide formal training in project management nor
provide a project controls manager or operations manager to assist lab
personnel assigned as Project Managers. Absent this background and
support, Fermilab project managers are left to fashion their own
individual approach to project management processes, organizational
structure and operational procedures (design, fabrication and assembly,
testing and QA); this contributed to the failures within the devices.
Typically, as a research lab, Fermilab draws upon its physicists to
staff key roles in project management. This is not an issue in and of
itself; however, as their expertise and primary focus is on scientific
technical matters. Project management procedures and systems are not
embraced and employed effectively in the project execution. This is
evidenced by the following:

-   Project management not fully understood or supported

    -   The review of the project documentation revealed a full set of
        documents such as the Project Execution Plan, Quality Assurance
        Plan, Budget, Schedule, etc. All of the correct terms were used
        in the development of these items; however, with the exception
        of the budget and schedule, the project documents were "put on
        the shelf" and not used as a guideline. The Project Execution
        Plan had the appearance of project management plan which
        contained appropriate guidelines, was compliant with DOE
        requirements, and was not applied once complete.

    -   Standard tools that one would expect to be used as part of a
        Project Execution Plan, such as risk management, were absent.
        Personnel interviewed were questioned about how risk was
        identified and abated. They stated that there was no risk
        analysis. This is especially poignant during the time when it
        was determined there would be no string test due to budget and
        schedule constraints. Additionally, a string test was never
        specified as a part of the US Project. CERN had planned from the
        beginning that the first complete assembly test would be when
        the magnet system was put in place in the LHC.

> The decision not to perform the string test may have been a good one
> as tests and processes are considered and abandoned as part of any
> project. The issue from the project management process and system
> perspective is the failure to actually determine the cost and schedule
> impact and, if it was determined that the constraint is true, to
> perform a risk analysis of that decision. The project should
> systematically consider the elements of risk and how to mitigate them,
> or make a conscious decision to accept the risk.
>
> Regarding the string test; items that should have been considered
> include understanding what information was going to be gleaned from
> the test if it was run, deciding whether there were any compensatory
> measures that could have been substituted for the test in a cost
> effective manner to give the project the same result, and the
> potential outcomes of not doing those things. Potential outcomes
> associated with failure could include safety and operational problems
> and denigration of Fermilab's technical prominence; leading to costly
> re-design, repair or replacement of components.
>
> In the case of the string test there were other things that could have
> been done in lieu of a physical test such as commissioning an
> integrated model that ties the quadrupoles and DFBXs into triplet
> assemblies. The act of building the model would have driven questions
> such as where the 20 bar pressure comes from and how it is applied --
> which may have led to the discovery that the dynamic load had not been
> considered.
>
> No such systematic review took place and it became widely accepted
> that the test could not be done due to budget and schedule
> constraints.

-   The project lacked integration from the onset. Key items were not
    considered and acted upon:

    -   Confusion about CERN versus Fermilab roles and responsibilities
        led to blurred lines when it came time to baseline the project.
        Responsibilities were not fully defined (e.g. determining who
        was responsible for design of CERN tunnel support systems for
        the IRs).

    -   Agreements did not include Fermilab review of test plans for the
        equipment once it was installed at CERN. Fermilab's
        responsibility ended when the equipment was received and
        inspected by CERN. By not reviewing the test plans, or being
        part of the commissioning and startup of the accelerator,
        Fermilab lost the opportunity to understand how their equipment
        was to be used and the commensurate operational parameters it
        would encounter. If Fermilab had reviewed the test plans it may
        have keyed them to consider the dynamic pressure loading.

    -   The organizational chart showed a systems integration manager
        that was initially filled, but was later vacated to assist LBNL
        in recovery of the DFBX project. The decision not backfill the
        position was made in response to budget constraints. In
        addition, there were mixed responses to describing the role of
        the integration manager which varied from a part-time position
        to integrate components between the LBNL, BNL, and Fermilab to a
        full-time position involved with integrating the work within the
        fence at Fermilab.

> The result is that no one in the project was questioning how these
> components would fit together or whether the integrated components, as
> a system, would meet performance specifications/operational
> requirements at CERN.
>
> Traditional projects of comparable size combine the integration and
> risk management roles so that there is an individual whose sole
> purpose is to be concerned with the integration of system components
> and assessing risk to the project.

-   The project did not execute a lessons learned program. Many of the
    participants were involved with Tevatron, upgrades and new
    construction at Fermilab, and the SSC yet there was no group
    discussion at the beginning of the project to develop a list of
    lessons learned from those projects for application to the LHC.

-   The project did not avail itself with expertise outside of Technical
    Division. There were experts in cryogenics in the Accelerator
    Division as well as modeling expertise in PPD. These assets could
    have been used for reviews or to augment Technical Division staff
    during times of heavy workload. As a result, there was no truly
    independent second checking of the work within the Fermilab fence.

*Distraction*

-   Cost and schedule control

    -   The cost elements of the schedule were developed with varying
        levels of basis; the fabrication portion was fairly well known
        as it was based on experience from the RHIC magnets at BNL while
        the R&D component was less developed.

    -   Fermilab, BNL, and LBNL were allocated 40/40/20% of the project
        budget respectively. The team could find no evidence that the
        project coupled the capability to perform the scope to the
        funding levels awarded the sites. In addition, in 1996 it was
        decided to de-scope the project to meet budget restrictions
        resulting in realignment of scope between the laboratories. At
        that time, the responsibility for the DFBX was shifted to LBNL
        who, in retrospect, was not qualified to design and build the
        component.

    -   It was approximately three and one-half years into the project
        before problems with the DFBX were acted upon. Project personnel
        had reported that the design was complete and ready to build,
        but was not. A key Fermilab employee was re-assigned to work
        extensively with LBNL, as well as the assignment of a new chief
        engineer at LBNL, to recover the project.

-   Management systems

    -   It was apparent that Fermilab senior lab management was not
        actively engaged in the project execution. Once the project was
        funded the PM was essentially on his own. He reported to the
        Fermilab Director who looks at projects from a Level 0 budget
        and schedule perspective, as is appropriate, and guidance from
        the head of the Technical Division focused more on staff
        resources. Without guidance or limitations, the only oversight
        fell to project reviews which, by their nature, are not detailed
        enough to discover the issues that caused the events. An
        experienced senior manager may have discovered that the design
        reviews were not adequate and directed the PM to commission a
        review to confirm design basis calculations and drawings. This
        did happen when the current Fermilab Director commissioned an
        in-depth design review of the suggested repairs of the Fermilab
        LHC components following the failure in the LHC. The project had
        again performed a summary review (PowerPoint®) when the Director
        stepped in and made the in-depth review a requirement.
        Participants in the review provided positive feedback that this
        type of review added value to the design process and that it
        should be used more regularly in future design efforts.

    -   There was no recognition of when the project transitioned from
        the R&D phase to the design-build phase. Because it was not
        recognized, R&D continued to occur well into the design-build
        phase causing changes to occur while the Technical Division was
        attempting to put systems together. A point in the project could
        have been picked where no more R&D inputs would be accepted to
        allow focus on production. At the same time, R&D efforts could
        have been directed to the LARP for future consideration and
        upgrade of LHC, once operational.

    -   Collaboration versus vendor relationships can have a negative
        affect if not managed properly. On the positive side,
        collaboration can result in the supplied components being closer
        to what is required by the customer. On the negative side,
        collaboration can lead to a less formal relationship where
        scope, schedule, and funding limitations can be ignored in favor
        of providing a better product. This project was more
        collaborative, and both of the above happened to some extent,
        neither of which presented a major problem for Fermilab or CERN.

> It is mentioned in this report because it was brought up by almost
> every person interviewed when discussing key milestones and
> deliverables. The point of those interviewed was that a project can
> have a collaborative or vendor relationship, but not both. The team
> disagrees.
>
> A project can be collaborative with certain key milestones agreed to
> by both parties. This is particularly valuable when the project is
> setting a date when all parties will understand performance
> specifications. This will have a positive impact because all parties
> will be able to plan resources to make that happen, which they can
> then work collaboratively to define. Without that occurring, the
> project can get to a point where one party is ready to fabricate their
> components and another is still in design, which ultimately impacts
> the project's ability to deliver a final, integrated product on time.

[]{#__RefHeading___Toc174459531 .anchor}**[Root Cause
Discussion]{.underline}**

[]{#__RefHeading___Toc174459532 .anchor}2)e.i.1 - Standards, Policies,
or Administrative Controls Incomplete

*Root Cause Statement*

Fermilab engineering management controls do not include codified,
standard design process requirements that include a systems integrated
design, design review, and documentation recording and archival process.
Instead, Fermilab relies upon individual contributors to obtain review
of design basis calculations and recognition of interface and
integration requirements. In both instances, the lack of documentation
and in-depth review resulted in critical design errors being missed
until the components were tested *in situ* at CERN.

*Root Cause Discussion*

If the controls had been in place to check and review design and view
the devices as an integrated system, the direct cause of an error in
design would more likely have been caught and would probably have not
made it through the process to fabrication and delivery. Put another
way, controls, also referred to as barriers, should be in place to catch
and correct issues such as the ones that happened during this project.

*Similar Issues*

Senior management interviewed at Fermilab and DOE drew the Team's
attention to similar experiences regarding the design and fabrication of
components and systems, one of which is presented in a summary format,
below. This, and other similar issues, leads the Team to conclude that
the issues encountered in the LHC project are not limited to this
project.

-   Main Injector Project

    -   The installed system piping wasn't what was specified (i.e. did
        not have full penetration welds)

    -   The vendor did not perform required QC checks (i.e. weld-joints
        weren't X-Rayed to verify a full penetration weld)

    -   Fermilab did not have quality assurance inspectors to verify
        welds at the point of fabrication/manufacture and did not demand
        to see the X-Ray films to verify the welds

    -   Fermilab performed hydrostatic test improperly

        -   Used tap water in a stainless steel system

        -   Left system in wet-layup with tap water

    -   Welds were attacked and leaked while in layup

    -   All welds had to be re-worked at an expense of over \$1.5M

        *Root Cause Conclusion*

The current competitive environment presents new challenges in the way
that projects are funded and executed.

-   Funding sources are limited and grants are more restrictive in terms
    of expectations and oversight requirements.

-   High-Energy Physics projects are large and costly resulting in an
    increased use of collaborative ventures between laboratories,
    academic institutions and international research organizations.

-   Government funding sources require increased scrutiny of the project
    execution to ensure each participating member of the collaborative
    venture meet its obligations:

    -   Performance criteria must be identified and met in accordance
        with project scope

    -   Systems and components must work in the intended operating
        environment

    -   Interface requirements must be included in design

    -   Risk must be understood and mitigated

    -   Projects should be managed within budget and schedule
        constraints.

Meeting the challenges of the current competitive environment, coupled
with the desire to host the ILC at Fermilab, warrants modification to
the way that projects are managed.

[]{#__RefHeading___Toc174459533
.anchor}**[Recommendations]{.underline}**

The following recommendations were derived from the cause sections,
above. They directly answer the issues discussed, and for the most part,
can be implemented in lieu of current requirements. The team has made
every effort to minimize administrative burdens.

In addition to the recommended actions, the Team strongly urges Fermilab
to commission a Process Improvement Team to develop guidelines for
Fermilab on how to improve its approach to executing both
inter-laboratory and international collaborative projects. It is the
Team's opinion that this type of relationship will be more prevalent in
the future and needs to be addressed to make Fermilab a success and a
serious contender for future international projects, such as the
International Linear Collider.

Finally, the Team wishes to put the recommendations into context. The
team was asked to perform a critical look at issues at Fermilab
regarding their performance. The issues discussed are comprehensive and
valid; however, there is a scalability issue to be recognized regarding
adopting the recommendations. It is based entirely upon the assumed
future of Fermilab. When deciding which of the recommendations to
follow, and to what level, the Fermilab strategic plan should be
addressed and a few pointed questions should be asked:

1.  Is the laboratory safe without adding this requirement? For the most
    part, yes although one could make the argument that an undiscovered
    design flaw could easily lead to adverse impact on personnel safety.

2.  Can Fermilab continue the way it has for the past 40 years? Yes, the
    process has worked so far and will probably work in the future
    although changing a few of the processes will result in reduced
    issues and rework and; therefore, more money available for science
    versus re-work.

3.  Can Fermilab count on the ILC or other large project coming here?
    No. The Fermilab processes will not endear the laboratory to
    international site selection committees if it continues to have the
    types and frequencies of project issues it has experienced.

> While a decision to retain its current practices will not result in
> Fermilab's immediate demise, it may result in the site being passed
> over in favor of others when it is time to decide where to host the
> next large accelerator project.

The following are specific recommendations the Team believes will have a
positive impact on the laboratory, both now and in future endeavors in
the international HEP community.

-   []{#__RefHeading___Toc174459534 .anchor}Project Planning

    -   Projects should include a standard kickoff process during the
        formation of the project which covers major elements, how they
        are going to be controlled and by whom, project issues and
        risks, and performance expectations. At a minimum, the meeting
        should be held over a period of days, but may expand to one or
        two weeks depending upon project size and complexity, as
        required to cover the following:

        -   Brainstorming the approach

        -   Identifying lessons learned from previous projects of a
            similar nature. A small investment in time up front will pay
            large dividends in avoided rework later.

            -   This should be an ongoing effort that continues to
                capture lessons learned for the current and future
                projects. The lessons learned should be archived and be
                available to other project personnel.

            -   To be effective, incidents and/or events during design,
                procurement, fabrication and testing should be
                accurately and thoroughly documented to capture the
                facts of the issue, what was done to immediately correct
                the issue, and what was changed to prevent recurrence.

        -   Discussing how risk is going to be identified, calculated,
            managed, and mitigated.

            -   Known issues, risks, and associated mitigating actions
                should be identified and tracked from the beginning of
                the project

            -   Identification of potential issues that can't initially
                be qualified or quantified should be noted for
                monitoring during the project execution.

        -   Identifying project constraints on the project in terms of
            budget, schedule, manpower, and known operational issues.

        -   Identifying, discussing, and assigning key project members
            roles and responsibilities:

            -   Project Manager and Deputy

            -   Finance

            -   Procurement

            -   Integration and Risk Management Manager

            -   Project Technical Lead

            -   QA

            -   Project Controls

        -   Discussing what inter-divisional participation will be
            sought to augment the project and how it will be managed.

        -   Developing test plans and identifying testing environments

        -   Documenting the known operational environment. This will
            lead to the identification of knowledge gaps or areas that
            will require additional work to develop, and should involve
            a considerable effort.

            -   This exercise should develop a list of performance and
                operational specifications with associated actions to
                fill the gaps.

> Project Controls - Project controls should be codified to describe how
> projects will be managed at Fermilab:

-   Budget and cost management should use a qualified Earned Value
    Management System to track and control costs.

-   The project should generate and archive the Basis Of Estimate (BOE)
    for all major project elements.

    -   Develop and maintain a resource-loaded baseline schedule

-   A risk-based change management process should be established that
    identifies variance threshold reporting and action requirements.

-   Contingency versus management reserve should be identified.
    Traditionally, management reserve is the portion of the project
    funding withheld from the project participants by the Project
    Manager that can be released for overruns and changes to estimated
    project elements. This should be separate from contingency funds
    which are typically assigned for changes in scope or timing of
    deliverables brought on by changes in schedule outside the PM's
    position.

-   Schedule management

    -   The schedule should contain hard schedule requirements (3-5% of
        project cost) that are inviolate. This will ensure that certain
        key elements are not deleted to improve the financial condition
        of the project.

        -   As a standard practice all project baseline budgets and
            schedules should include specific activities and resources
            for engineering review, testing and other QA activities.
            These activities and budget items should be independent of
            the project contingency. Recommend at least 3-5% of the
            project budget be typically allocated to these sets of
            activities.

        -   The schedule should also contain a formal break point
            milestone transitioning from R&D to production negotiated
            between the supplier or owner of the ultimate product and
            the vendor/collaborator

-   Project documentation

    -   Risk analysis: a formal risk analysis should be conducted and
        documented at the conclusion of conceptual design and the
        mitigation strategies incorporated in the project baseline
        budget and schedule (in accordance with DOE O 414.1C). The risk
        analysis should be updated and reviewed at each subsequent
        design review milestone and at least quarterly during the
        fabrication, installation and commissioning phase.

-   Systems Integration Role: For any complex or collaborative project
    the engineering/PM organization should budget and staff a specific
    role for ongoing systems integration. This role should continually
    oversee how the technical managers are addressing integration
    issues, conduct regular systems reviews, highlight configuration or
    requirements changes that each technical designer needs to keep
    current, and coordinate sharing of integration issues and solutions
    among all team members. This person may also serve as the risk
    manager on smaller projects.

    -   There needs to be a continual updating and communication to all
        team members of changes to the functional requirements as
        designs progress or project conditions change.

    -   Parallel to the systems role, on international projects, there
        should be a code conformity document to guide the development so
        that the devices function as anticipated in under the standards
        and performance codes of the operational environment

```{=html}
<!-- -->
```
-   []{#__RefHeading___Toc174459535 .anchor}Design and Engineering
    Process

> The design and engineering process at Fermilab currently follows a
> distributed model where engineers and scientific personnel fulfilling
> engineering roles are assigned to separate divisions based upon need.
> This is in contrast to some laboratories where the engineering
> function is centrally controlled and engineering support is matrixed
> to other organizations as needed to support their projects.
>
> Both approaches have their individual merit; the two primary themes
> identified are the lack of control and integration with the
> distributed model and the lack of attention for smaller, shorter-term
> projects under the centralized model. The Team did not evaluate which
> would work better at Fermilab; however, it does recommend that a
> Fermilab Chief Engineer position be created. The Chief Engineer should
> be the engineering process owner and publish a centrally-controlled
> design/engineering manual and implementing procedures that govern how
> that function is controlled at Fermilab, which would be mandatory for
> all engineering processes at Fermilab. The Chief Engineer could also
> be called upon by the OPMO and Project Management to select qualified
> individuals and lead design/engineering reviews.
>
> The Design/Engineering Manual should contain elements that govern
> reviews, data archive, process, etc.:

[]{#__RefHeading___Toc174459536 .anchor}*Design Reviews:*

-   Templates and checklists for what materials must be produced by
    design teams in advance of each review, to include calculations,
    design bases etc. These will provide an internal QA tool for the
    engineering team and provide the required materials to allow
    comprehensive independent technical reviews. These templates should
    be incorporated in a design review procedure. The checklists will
    help ensure that all elements of the design are reviewed including
    inter-disciplinary checks and all specifications, boundary
    conditions, testing requirements, and operational environments have
    been considered and appropriately applied.

-   Design reviews should include technical experts from outside the
    team and in certain cases outside of the lab. This will assist in
    providing fresh independent assessments of completeness. A budget to
    pay for this reviewer's time from outside the team or from outside
    experts should be established so cost is not a constraint. A team of
    on-call resources to assist in these reviews should be assembled
    through contract resources ahead of time so they can be engaged
    quickly as required.

-   Review Formats: The design reviews should include sufficient time
    for on-board review of specification/drawings assumptions etc. The
    review team should have time to review these materials in an
    independent manner. The review team should prepare its own set of
    written comments and suggestions and then be engaged subsequently to
    review how the design team addressed or incorporated the comments.
    The "presentation" or PowerPoint® review portion should be limited
    so as not to conflict with the time required to perform technical
    reviews (or alternatively advance material such as in a PP format
    could be emailed to the review team in advance of the review meeting
    to facilitate their being aware of progress since the last meeting
    in advance of the current meeting).

-   The current changes to the functional requirements and
    specifications should also be a component for each technical design
    review to verify that the current design is consistent with the
    current requirements.

    []{#__RefHeading___Toc174459537 .anchor}*Design Process Management*

-   Establish a configuration control system for design efforts

-   Define design basis archive requirements and establish a central
    library where such information is stored and available for future
    use

-   Define the requirement for design basis cross-checks. This includes
    calculations, drawing generation, and interface requirements
    reviewed both internally and between sub-element projects designing
    specific, but separate, components.

-   Archive R&D design specification development and control. Include
    items such as ideas tried and abandoned, and why; shop floor
    changes, reason, and impact; and results of tests against a
    prescribed (assumed) specifications.

-   Define the change management process using a risk-based graded
    approach that defines who can authorize changes and under what
    circumstances.

-   Develop a process that identifies critical designs and the extra
    requirements to control them

```{=html}
<!-- -->
```
-   []{#__RefHeading___Toc174459538 .anchor}Training

    -   The Team recommends that formal training for Project Managers be
        developed or procured. This training will be helpful to those
        managing projects at Fermilab so they can learn to manage the
        projects versus being led by them.

    -   Develop orientation for engineers or employees performing the
        engineering process so they understand the processes employed at
        Fermilab

-   []{#__RefHeading___Toc174459539 .anchor}QA Program

    -   The lab/engineering organization should prepare and compile an
        ongoing electronic database of "lessons learned" from
        engineering reviews, testing results and other failure lessons
        that are available to all personnel across all disciplines.
        (Analogous to safety incident reports/near miss reports etc.)
        This will improve the sharing of information across divisions,
        labs and project teams and provide a useful tool for training of
        engineers and PM teams. This is increasingly important as more
        and more projects will be collaborations among divisions,
        laboratories and universities.

    -   All project budgets and schedules should include specific
        activities for quality assurance such as design reviews, factory
        inspection, system testing etc. and these budgets and schedules
        should be distinct from "contingency" activities. This will
        elevate these activities in importance and require any deviation
        from them for budget and schedule considerations to be more
        formally considered. (see budget and schedule).

    -   Fermilab is currently developing a Quality Assurance Plan and
        implementing documentation. This should include elements that
        direct quality implementation using a risk-based graded
        approach. It should also include the adoption of a Quality
        Assurance Inspection program that defines how inspections will
        be conducted and a qualification program to standardize how
        these inspections are performed. The qualification program
        should be performance based that includes a combination of
        classroom and on-the-job training against a qualification
        standard managed by the senior (QAI Level III) qualified Quality
        Assurance Inspector.

    -   The fabrication process traveler should include a review by the
        resident QAI -- III to concur with QA stop and hold points.

        []{#__RefHeading___Toc174459540 .anchor}**[Supporting
        Documentation]{.underline}**

The Team used a combination of document reviews, collection of
statements, and interviews in the conduct of the root cause analysis.
Documents containing the results of the process review are provided as
supporting documentation to Fermilab at the conclusion of the review and
include:

-   Statements by project personnel

-   Interview notes

-   Functional area review discussion

-   Question matrix

-   Ishikawa diagram

-   Event and Causal Factor Charts

-   Timeline

**Attachments**

[]{#__RefHeading___Toc174459541 .anchor}**Attachment 1, Page 1 -- Root
Cause Analysis Flowchart**

![](media/image1.wmf){width="9.0in" height="5.872222222222222in"}

**Attachment 1, Page 2 -- Root Cause Analysis Flowchart**

![](media/image2.wmf){width="5.892361111111111in"
height="8.741666666666667in"}

[]{#__RefHeading___Toc174459542 .anchor}**Attachment 2 -- Personnel
Interview List**

***[BNL]{.underline}***

Mike Anerella, Chief Engineer

***[CERN]{.underline}***

Ranko Ostojic, Specialty Magnet Manager (current)

Tom Taylor, Specialty Magnet Manager (former)

***[DOE]{.underline}***

Bruce Strauss, Program Manager

Jim Yeck, Project Manager

Pepin Carolan, Project Manager

***[Fermilab]{.underline}***

Barbara Brooks, Workforce Development & Resources Section Deputy Dept.
Head

Bob Kephart, Technical Division Head (2002-2005)

Cindy Conger, Chief Financial Officer

Connie Trimby, Laboratory Financial Planning Manager

Dean Hoffer, Office of Project Management Oversight

Ed Temple, Office of Project Management Oversight Head

Fred Nobrega, Quadrupole Cold Mass Project Engineer

Greg Kobliska, Technical Division Procurement Manager

Jamie Blowers, Technical Division QA

Jay Theilacker**,** Cryogenics Dept. Head, Accelerator Division

Jim Kerby, Fermilab LHC Accelerator Project Manager

Jim Strait, US LHC Accelerator Project Manager

John Peoples, Former Fermilab Director

John Zweibohmer, Technical Division Deputy Procurement Manager

Mike Lamm, Quadrupole Integration and Testing Scientist

Paul Czarapata, Accelerator Division Deputy Head

Peter Limon, Former Technical Division Head

Phil Pfund, US LHC Accelerator Engineering Manager

Roger Bossert, Quadrupole Magnet Cold Mass Engineer

Steve Holmes, Associate Director for Accelerators

Tom Nicol, Quadrupole Cryostat Project Engineer

Tom Page, Quadrupole Cryostat and Interconnect Engineer

Tom Peterson, Quadrupole Cryogenics Engineer

***[LBNL]{.underline}***

Joseph Rasson, DFBX Project Engineer

[^1]: TapRoot® is an Incident Investigation System which provides teams
    with a procedure and techniques to perform an in-depth review of an
    incident or event. The system looks for both human performance and
    equipment causes of incidents or events and is flexible to allow
    review team leaders to select only the techniques that are
    applicable to the event being reviewed.

[^2]: An Ishikawa diagram, commonly referred to as a fishbone diagram,
    is a visualization and knowledge organization tool. It allows the
    users to collect the ideas of a group in a systematic way that
    facilitates the understanding and ultimate diagnosis of a problem.
    The diagnosis leads the group in a systematic way to discover the
    root, direct, and contributing causes of a problem.