T. Dombeck

9/11/02

**FERMILAB ACCELRATOR CHAIN VULNERABILITIES**

**[Summary]{.underline}**

A study was undertaken to answer questions raised by Fermilab management
about the vulnerability of the accelerator chain to the failure of one
or more critical elements that might put the physics program out of
commission for a long period of time. In this report we limit the study
to those components that would jeopardize Collider operations for a
period of three or more months. This question was posed to those
responsible for the development, operation, and maintenance of the
facility. There were tens of critical components identified whose
failure might pose such a threat to operations. These items are
summarized in this report, along with recommendations that might
mitigate the possibility of a failure.

Historically, the Fermilab accelerator chain has had a few such
catastrophic failures that occurred during the early decades of
operation. A substation power transformer was lost in 1985 at which time
Fermilab was running only external beams \[1\]. Though the repair
required about eight months, the physics program at that time was able
to limp along, but with much reduced capability. Another catastrophic
failure during the first decade of Tevatron running resulted in the loss
of thirty-three superconducting dipole magnets in the single year of
1989 \[2, 3\]. Such a loss occurring at this time would virtually
exhaust the pool of spare magnets jeopardizing future running. The cause
of these particular catastrophic failures was investigated and their
sources have been addressed to ensure they do not recur.

The current study has uncovered an additional set of components that
might cause catastrophic downtimes due to failure. A number of these
critical elements lay in the oldest parts of the accelerator chain,
namely the Linac, the 8-GeV Booster, and their associated transfer
lines. These Proton Source systems have performed remarkably well over
their thirty years of operations. In the case of the Booster, there is
no reason to doubt it will continue to perform at its current repetition
rate of about 2 Hz. However, the proposed increase in pulse rate from 2
to 12 Hz to meet the demands of the Mini-Boone and NUMI experiments,
will place a strain on the aging equipment.

Furthermore, there are some components that were not designed for the
increased Booster repetition rates, for instance, the four orbit bump
magnets. There have not been any failures in these magnets over the ten
years of their operation, and there are two spares, however there are
concerns of heating problems at the highest repetition rates. Tests
indicate that the magnets may perform appropriately up to 7.5 Hz and
this would meet the requirements for the Tevatron and Mini-Boone. Higher
repetition rates (\~12 Hz) needed to include NUMI, may require a
modification of the Booster injection system \[4\].

In this report, vulnerabilities due to one-of-a-kind components are also
discussed. Such items appear throughout the accelerator complex. Though
reliability has been good, some of these items do not have spares, such
as the 400-MeV chopper at the end of the Linac. Other specialty items,
such as kicker magnets, have single spares, but the lack of availability
of replacement parts could pose a problem. To limit the scope of the
present study, some one-of-a-kind components were not considered, for
example a failure in the focusing horns in the NUMI or Mini-Boone
beamlines, or failures in the experimental detectors. The repair of such
devices might require much time to implement and put those experiments
out of commission. However, such components were not considered because
the Collider accelerator complex would not necessarily be affected by
their failure.

In some cases, having spares would be prohibitively expensive, such as
the Linac accelerating cavities. Creating spares would require much
resource due to the number of unique structures down the length of the
machine. Furthermore, experience at other laboratories, ANL \[5\] and
LAMPF \[6\], have shown that accelerating cavities rarely experience
catastrophic damage even after many decades of operation. Even though
replacing a damaged structure would require many months of downtime,
building an inventory of replacement parts would seem unwarranted.

There are other one-of-a-kind electrical devices that could stop
operations where it would be less expensive to provide a spare. Some of
these components are thirty years old, such as the 345-KV MSS
switchgear, and are beyond the Laboratory\'s capability to repair
in-house. In other cases, the expertise to fix the item has dissipated,
such as the low-level rf for the 8-GeV Booster. Presently, estimates to
repair or replace these items cover a range of potential downtimes and
may constitute a risk to operations. Other one-of-a-kind devices, such
as pickups in the Accumulator stochastic cooling system, are not as much
of a risk. In a crisis where one of these units is lost, it is believed
that a work-around or a replacement could be devised within a few weeks.

There are a few specialty components where multiple failures might
jeopardize operations because only one spare exists. In this category
are the low-beta magnets, the low-beta power supplies, and the Main
Injector power supply transformers. In each case, replacements require a
special order with a long-lead time. Procuring additional spares would
be advisable. However, as they are moderately expensive, a staged
approach might be possible. For instance, the greatest risk for the Main
Injector supplies occurs when the NUMI experiment is operational, so for
the immediate future the emphasis could be to provide an additional
spare for the Tevatron low-beta supplies.

Another example of high-risk specialty items, are the kicker magnets in
the Main Injector and in the Tevatron. In general, a failed kicker
magnet could be quickly repaired except for one critical component, the
long ceramic insert. Except for the single spares on hand, there is
currently no source available to fabricate ceramic replacements. In view
of this problem, the handling of the spare kickers should be placed
under particular scrutiny to prevent inadvertent breakage of these key
components. A long-term solution to obtain ceramic inserts should also
be considered.

There are a number of infrastructure items that might produce long down
periods if allowed to progress to the point of failure. Among these are
the silting of the cooling ponds around the Tevatron and the
degeneration of the wooden power poles. These items are inspected on a
regular basis, and there is a progressive remedial plan worked out,
though sufficient funding has not been identified as yet to complete the
work.

Finally, there are staffing concerns at all levels throughout the Beams
Division accelerator groups. The general feeling is that the complexity
of the system has grown in the last decade with the addition of the Main
Injector, the Recycler and various external beam experiments. The plans
are to expand the program further with B-TeV and a 120-GeV beamline.
There has not been a corresponding increase in personnel and has
resulted in more reliance on contract labor. In some cases there has
been a loss of particular expertise due to attrition. This could result
in accidents due to \"human error.\" Such was the case last year when a
technician unfamiliar with the Tevatron components inadvertently
connected a water cooling line to a He feed line resulting in the
contamination of ten superconducting components. The results were not
disastrous in this case, however, lack of proper personnel could make it
difficult to recover from a large-scale failure in a timely fashion,
thus jeopardizing the core physics research program of the Collider.

**[Introduction and Methodology]{.underline}**

The definition of what constitutes a critical component whose failure
could shut down Fermilab operations is intimately connected to the
resulting downtime that can be tolerated. The number of such components
increases dramatically, probably exponentially, with the inverse of the
tolerated downtime. For instance, there are probably thousands of such
items that might cause a one-week downtime. Fortunately, each such
failure has a small probability of occurring and this situation is best
analyzed from a machine reliability standpoint \[8\].

On the other hand, there are perhaps one hundred such critical
components that might cause a one-month downtime, tens of items causing
a three-month downtime, and only a few items that might cause a
six-month downtime. As one month is on the order of the usual downtime
for routine maintenance, a failure of this magnitude would probably
result in the next scheduled maintenance period being advanced.
Therefore, in this study we limited consideration to those items that
might cause extended downtimes of three months or longer.

That said, it is difficult to assess if a certain component failure
would indeed result in a downtime of three or more months. Facing such a
disaster, the Laboratory would muster its forces to provide workarounds
to bring operations back on line as soon as possible. On the other hand,
it is equally difficult to analyze how a particular failure might
cascade and generate related failures that spread throughout the
complex, e. g., a power failure that ruins electronic circuits or
computing files in other systems resulting in months of trouble
shooting. Therefore, the philosophy in this report is to err on the
conservative side, if a three-month downtime were believed to be
possible for any particular component, it was put on the list.

This report contains a set of tables presented by accelerator type
and/or subsystem with items identified by accelerator personnel that
were deemed to pose a risk to operations. Comments are given on the
current status of the identified components. A risk factor (high, medium
or low) is assigned based on the perceived probability for a failure,
though no attempt has been made to quantitatively analyze the
probability of a failure occurring. Only in a few cases, such as
superconducting dipoles, has there been a sufficient sample to arrive at
meaningful conclusions. Finally, a discussion follows with
recommendations to mitigate the effect of a potential failure for the
highest risk items.

A summary is presented at the end of the report of a mutual study
performed between Beams and Technical Division personnel. This study
addressed the status of spares and the general question of performing
repairs on large components such as conventional and superconducting
magnets, as well as Tevatron spool pieces. These components require
large-scale in-house maintenance facilities. In essence it was felt that
there were sufficient spares on hand for most of the accelerator chain.
There were exceptions. For instance, there are sufficient spares for the
gradient magnets in the Booster, however the operational status of these
spares is not confirmed. There are also a few specialty magnets, such as
the kickers in the Main Injector and Tevatron, that have a spare, but
fixing a broken device might not be possible. For other one-of-a-kind
magnets, it was felt that even with a need to bring old tooling out of
\"moth balls,\" repairs could be achieved in the allotted time with the
initiation of a crash program at the time of the failure.

Finally, there could be catastrophic losses due to fires or floods that
might affect large parts of the accelerator complex. Disasters of this
sort are better studied from an historical perspective along with the
standard equipment installed to mitigate losses. Therefore, these are
not discussed in this report. On the other hand, there are a number of
potential risks for \"human error\" accidents, some of which are
discussed in this report. For example, there are cases where both the
piece of equipment and its spare are stored in the same location. This
practice might jeopardize the recovery of operations after a fire. The
vulnerability due to such procedural concerns are not discussed in any
detail in this report, however, it might warrant a future investigation
to determine what mitigating actions might be undertaken.

**[Front End Source, Linac, and Associated Beamlines]{.underline}**

  ----------------------------- ----------------------- -------------------------- -------------------------- ------------------------------
  **[Component]{.underline}**   **[No.]{.underline}**   **[Spares]{.underline}**   **[Risk]{.underline}\***   **[Vulnerability
                                                                                                              Comments]{.underline}**

                                                                                                              

  Cockroft-Walton               2                                                  L                          Aging equipment, but two
                                                                                                              sources in operation, thus
                                                                                                              providing a spare (See
                                                                                                              discussion below.).

                                                                                                              

  750-kV, 200-MHz Buncher       1                       0                          L                          One of the oldest pieces of
  Cavity                                                                                                      equipment in the accelerator
                                                                                                              chain (See discussion below.).

                                                                                                              

  200-MHZ Structures and        5                       0                          L                          Some of the oldest pieces of
  In-Tank Quads                                                                                               equipment in the accelerator
                                                                                                              chain (See discussion below.).

                                                                                                              

  #7835 Amplifier Tubes (200    5                       1                          H                          Failure rate is about 3.5/yr.,
  MHz)                                                                                                        7 in rebuild, single vendor
                                                                                                              (See discussion below.)

  Modulator Switching Tubes     15                      40                         M                          Failure rate is about 7/yr.,
  (200 MHz)                                                                                                   single vendor (See discussion
                                                                                                              below.)

                                                                                                              

  805-MHz Structures and        28                      0                          L                          10 yrs. old (See discussion
  In-Tank Quads                                                                                               below.).

                                                                                                              

  400-MeV Chopper               1                       0                          L                          Required to run booster, no
                                                                                                              backup, but reliability high
                                                                                                              so not deemed risk at this
                                                                                                              time (See discussion below.)

  400-MeV Spectrometer Magnet   1                       0                          L                          No backup, but magnet has
                                                                                                              spare coil capacity and so not
                                                                                                              deemed risk at this time

                                                                                                              
  ----------------------------- ----------------------- -------------------------- -------------------------- ------------------------------

\*Risk of failure deemed H=high, M=medium, and L=low.

[The Front End Source and the 200-MHz Components]{.underline} are some
of the oldest pieces of equipment in the accelerator chain. They must be
operational in order to have any protons available for the physics
program. They are expected to operate with high availability, \>97%, and
have done so for many years. However, signs of aging are appearing, such
as the modulators and the Amplifier Tubes, and the lack of vendors to
make replacement parts.

The Front End Source situation was recently summarized in a talk
presented by Bob Webber to the Laboratory Directorate \[9\] along with a
proposal to replace the oldest items with more up-to-date components,
such as an rfQ and 402-MHz accelerating cavities. The cost of such an
upgrade is estimated to be about \$27.5M, however, there are a number of
reasons to consider such an upgrade, including a promise of a five times
brighter proton beam. Even with the most optimistic timetable for the
upgrade to take place, there would be a fairly long period of at least 6
years using the present equipment, in which backup 200-MHz parts must be
maintained.

The Linac already runs continuously at 15 Hz, so the additional strain
on operations imposed by the MiniBoone and NUMI experiments, will not
affect the Linac requirements. Therefore, the average failure rates
observed for Linac components are not anticipated to change in the near
term and the required number of spare parts can be determined with a
reasonable accuracy. For instance, the accelerating cavities and the
quadrupoles that reside in the tanks can be expected to operate with
little risk of failure as experience at other longer-running Linacs has
shown \[5-6\]. However, the external rf power generation system requires
an adequate flow of replacement parts. Maintaining this flow probably
poses the greatest risk to continued Linac operations at this time (see
below).

[Amplifier Tubes (#7835, 200 MHz)]{.underline} are required for
generation of rf power for the first section of Linac cavities, with
five in operation at any given time. Experience with these tubes \[10\]
indicates that the failure rate is about 3.5 tubes per year. Currently,
there is one \"good\" spare and one \"bad\" spare in the inventory.
However, there is one vendor, Burle Industries \[11\], that builds new
amplifier tubes and rebuilds failed ones. It requires about nine months
to complete one tube. In recent times there have also been concerns that
the life expectancy of the rebuilt tubes may be less than hoped \[10\].

The ideal situation would be to have eight spares on hand \[9\]. This
would allow some recovery time in case of the possible loss of the sole
vendor, or a rash of replacement tubes that do not meet specifications.

*Recommendation*: There is excess capacity at Burle Industries at this
time to double the rate of delivered Amplifier Tubes \[11\]. Though the
additional rate would be expensive, it would permit the inventory of
spares to rise to a sufficient level to avert long downtimes due to tube
rebuilds.

[Modulator Switching Tubes (200-MHz)]{.underline} fail at the rate of
about seven per year, out of the fifteen in operation at any given time.
There was an order of fifty tubes from the single vendor, out of which
there are now about forty tubes in the spares inventory. There has been
a successful rebuild of a failed tube.

*Recommendation*: The capability to rebuild failed Modulator Switching
Tubes should be encouraged.

[400-MeV Chopper]{.underline} is required for the operation of the
Booster. There is no spare chopper or backup to its driving electronics,
though spare cards exist. Furthermore, there has been the loss of
expertise recently. The effect of the loss of the chopper has not been
ascertained, though it does clean up the Linac pulses to reduce beam
losses in the Booster.

The chopper is an electrostatic device that requires fine alignment of
the plates. There are few elements that can break if left as is,
however, there would be a risk of upsetting the device if it were moved.

**[8-GeV Booster and MI-8 Beamline]{.underline}**

  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------
  **[Component]{.underline}**   **[No.]{.underline}**   **[Spares]{.underline}**   **[Risk]{.underline}**   **[Comment]{.underline}**

                                                                                                            

  Gradient Magnets              98                      8                          L                        Operational status of the
                                                                                                            spares is not known (See
                                                                                                            discussion below.)

  Orbit Bump Magnets            4                       2                          H                        High rep rate will cause
                                                                                                            thermal problems (See
                                                                                                            discussion below.)

                                                                                                            

  Kicker Magnet                 9                       1                          L                        Replacement ceramic inserts
                                                                                                            under fabrication (See
                                                                                                            discussion below.)

                                                                                                            

  Low-Level rf System           1                       0                          H                        Outdated electronics (See
                                                                                                            discussion below.)

                                                                                                            
  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------

[Ring Gradient Magnets]{.underline} cycle at 15 Hz so the increased
repetition rate of the Booster for the neutrino experiments should not
stress these components more than under present operations. Furthermore,
there have not been any failures in these magnets over many decades.
There are spares, but they have not been checked for many years, so
their operational readiness is not guaranteed.

*Recommendation*: Identify where all spare gradient magnets for the
Booster ring are stored and power test a few of them to ensure their
viability.

[The Orbit Bump Magnets]{.underline} were installed ten years ago when
the injection energy was increased to 400 MeV. They were originally
designed for low repetition rates appropriate for Tevatron operations
(\~2 Hz) and have poor heat transfer making cooling difficult.
Mini-Boone requirements will increase the repetition rate to 6.5 Hz and
the heat generation will increase accordingly. A spare orbit bump magnet
was tested at 7.5 Hz operation for 36 hours and though the temperature
rose significantly, there was no failure. This might allow operations
for the Mini-Boone experiment with the currently installed magnets.

The addition of NUMI operations will increase the repetition rate
further to about 12.5 Hz with a corresponding increase in temperature.
There is a risk is these magnets will fail at the higher repetition
rates. There are two spares, but they will not suffice in the case of
the added requirements.

As part of the Proton Driver study, there were suggestions presented for
improved injection lines that would operate at 15 Hz thus alleviating
the orbit bump problem \[4\]. These upgrades might be expensive and
require downtime to install.

*Recommendation*: Studies should continue on ways to improve cooling of
the present orbit bump magnets. However, a parallel study should be
initiated on improving the Booster injection line to accommodate 15 Hz
operations.

[Kicker Magnets]{.underline} in the Booster system are smaller than
those in the Main Injector or Tevatron. They do not suffer the same
problems of replacement parts, such as the ceramic inserts. (See general
discussion on Kickers in the Main Injector section.) Those needed for
Booster operations can be fabricated on the Laboratory site.

[The Low-Level rf System]{.underline} is old and was created from
diverse modules. The expertise to maintain it has dissipated. There are
replacement electronics cards.

*Recommendation*: The backup modules should be checked to ensure their
operational readiness in case of a failure. The long-term solution would
be to replace the system with more up-to-date electronics that matches
other systems throughout the accelerator complex.

**[Main Injector and Associated Beamlines]{.underline}**

  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------
  **[Component]{.underline}**   **[No.]{.underline}**   **[Spares]{.underline}**   **[Risk]{.underline}**   **[Comment]{.underline}**

                                                                                                            

  Harmonic Filter Damping       2                       1                          H                        Requires a double failure, but
  Resistors                                                                                                 could occur in either Main
                                                                                                            Injector or Tevatron (See
                                                                                                            discussion of Power Grid in
                                                                                                            Site Infrastructure section.)

  Dipole PS Transformers        12                      1                          H                        These are long-lead time items
                                                                                                            that could reduce the physics
                                                                                                            program for up to 6 months
                                                                                                            (See discussion below.)

  Quad PS Transformers          6                       1                          H                        These are long-lead time items
                                                                                                            that could reduce the physics
                                                                                                            program for up to 6 months
                                                                                                            (See discussion below.)

                                                                                                            

  Main Injector Quadrupoles     269                     26                         M                        Had a number of failures, but
                                                                                                            able to keep up with
                                                                                                            replacement rate so far (See
                                                                                                            the section on the TD/BD Study
                                                                                                            on Magnet Spares.)

                                                                                                            

  Kicker Magnets                6                       1                          L                        One spare for each type (See
                                                                                                            the general discussion below.)

                                                                                                            
  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------

[Dipole and Quadrupole Power Supply Transformers]{.underline} have not
suffered any failures to date, however, special fabrication would
require about six months to replace these components. At present there
is one spare supply each for the dipole and for the quadrupole magnets.
They cost about \$150k and \$80k respectively to replace. The primary
risk would involve a double failure that would reduce operations, but
not necessarily shutdown the Collider program, as the Main Injector
could be run at a reduced repetition rate to compensate. However,
following a double failure, it might not be possible to run the NUMI
experiment.

*Recommendation*: Given the long-lead time, begin the procurement of two
additional spare power supply transformers, one for the Main Injector
dipole magnets (\$150k) and one for the quadrupole magnets (\$80k), to
have spares arrive when NUMI becomes operational.

[Kicker Magnets]{.underline} occur in various parts of the accelerator
complex: Booster, Main Injector, Accumulator, Recycler, and the
Tevatron. There is one spare for each type. Though the magnets suffered
some \"infant mortality,\" but in recent years there have been no
failures.

Responsibility for the kicker magnets is currently under discussion to
be transferred from Beams Division to Technical Division. Repairing a
damaged kicker magnet is generally not considered to be difficult, but
there is one critical component, the long ceramic inserts for the Main
Injector and Tevatron magnets. The sole vendor has stopped making
ceramics. In case of failures in these components, it is not clear at
this time how replacement inserts can be manufactured, though some
thought has been given to alternative materials. It should be pointed
out that the Booster kickers do not suffer from this problem because the
ceramic inserts are smaller and more readily available.

The difficulty in replacing the inserts places a special onus on the
handling of the spare magnets to ensure the ceramics are not damaged
inadvertently. Most of the kicker spares are stored in the F0 area, but
others are stored in areas around the Laboratory site.

*Recommendations*: The responsibility for the kicker magnets should be
resolved. The spares should be collected together in a secure location
to ensure proper handling. Work should continue to find a replacement
material for the long ceramic inserts, or a new vendor found.
Laboratories around the world should be poled to see if they have
developed sources for these critical components.

**[P-Bar Source, Accumulator and Associated Beamlines]{.underline}**

  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------
  **[Component]{.underline}**   **[No.]{.underline}**   **[Spares]{.underline}**   **[Risk]{.underline}**   **[Comment]{.underline}**

                                                                                                            

  Core Stochastic Cooling       1                       0                          L                        Pickups and kickers have no
  System                                                                                                    spares (See discussion
                                                                                                            below.).

                                                                                                            

  Kicker Magnets                4                       1                          H                        Spare for each type (See
                                                                                                            general discussion for Kickers
                                                                                                            in the Main Injector section.)

                                                                                                            
  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------

[Core Stochastic Cooling System]{.underline} components have been very
reliable since the early days of operations. There were a few failures
at the beginning, but these were easily repaired. Over many years of
operations, there have been no known failures inside the rf tanks,
similar to the experience with the Linac cavities. If there were a
failure of the pickups or kickers, it is anticipated these could be
repaired or replaced in less than three months. Therefore, maintaining a
pool of expensive spares is not deemed necessary.

Other rf elements in the p-bar cooling system have multiple tanks. If a
failure occurred in one of these, it is believed that the source could
be made to work utilizing only the working modules.

**[Recycler]{.underline}**

  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------
  **[Component]{.underline}**   **[No.]{.underline}**   **[Spares]{.underline}**   **[Risk]{.underline}**   **[Comment]{.underline}**

                                                                                                            

  Gradient Magnets              324                     8                          L                        Concerns about long-term
                                                                                                            depolarization of permanent
                                                                                                            magnets (See discussion
                                                                                                            below.)

  Specialty Magnets             8                       3                          L                        Concerns about long-term
                                                                                                            depolarization of permanent
                                                                                                            magnets (See discussion
                                                                                                            below.)

  Quadrupole Magnets            92                      0                          M                        All spares were used to modify
                                                                                                            ring lattice (See discussion
                                                                                                            below.)

                                                                                                            

  Kicker Magnets                2                       1                          H                        Spare for each type (See
                                                                                                            general discussion for Kickers
                                                                                                            in the Main Injector section.)

                                                                                                            
  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------

[Gradient, Quadrupole, and Specialty Permanent Magnets]{.underline} in
the Recycler ring and beam lines might lose magnetization due to long
term radiation and temperature effects. The magnets are compensated for
short-term temperature variations, and the ferrite material is also
known to have excellent radiation resistance. The long-term field
variations have been measured \[12\], and continue to be monitored in a
few magnets over long time periods. The results have indicated a 0.35%
loss in magnetization is expected over an estimated twenty-year
operating period. Therefore, the long-term loss in magnetization is not
considered to be a risk.

If there were a noticeable depolarization in a set of Recycler magnets,
they would have to be taken apart, the bricks remagnetized, and the
magnets reassembled. This could be accomplished at the rate of three
magnets per day given the availability of equipment and trained
personnel. Currently, the magnetizer is installed in IB2 and it is
estimated that reconstituting the permanent magnet factory could be
accomplished in a short time with a concerted effort.

[Quadrupole Magnet Spares]{.underline} have been used to modify the ring
lattice exhausting the pool. However, this is not considered to be a
major threat to Collider operations unless an accident occurred, i. e.,
a forklift damaged a large section of the ring. The Mini-Boone beamline
uses quadrupoles of the same basic design as the Recycler. There are two
spare magnets for this beamline that could be retrimmed within a few
weeks for use in the Recycler.

**[Tevatron Ring Components]{.underline}**

  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------
  **[Component]{.underline}**   **[No.]{.underline}**   **[Spares]{.underline}**   **[Risk]{.underline}**   **[Comment]{.underline}**

                                                                                                            

  Harmonic Filter Damping       2                       1                          H                        (See discussion on Power Grid
  Resistors                                                                                                 in Site Infrastructure
                                                                                                            section.)

                                                                                                            

  Kicker Magnets                4                       1                          H                        Spare for each type (See
                                                                                                            general discussion on Kickers
                                                                                                            in the Main Injector section.)

                                                                                                            

  Superconducting Dipoles       827                     35                         L                        Two different types (See below
                                                                                                            and the section on the TD/BD
                                                                                                            Study on Magnet Spares.)

  Superconducting Quads         212                     31                         L                        Eight different types. (See
                                                                                                            below and the section on the
                                                                                                            TD/BD Study on Magnet Spares.)

  Spool Pieces                  186                     38                         L                        Fifteen different types. (See
                                                                                                            the section on the TD/BD Study
                                                                                                            on Magnet Spares.)

                                                                                                            
  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------

[Superconducting Dipole and Quadrupole Magnets]{.underline} had original
design criteria for a twenty-year life expectancy. The Tevatron is
pushing this limit, however, the original criteria were based on the
more stringent requirements of the machine ramping at higher repetition
to service the fixed target program. Collider requirements are relaxed
in this regard, and it is doubtful that a fixed target Tevatron program
would be re-instituted. Furthermore, if life expectancy problems start
to arise, they would most likely manifest themselves through a continual
series of incidents whose frequency began to increase.

*Recommendation*: Assign responsibility to track and assess the
frequency of Tevatron component failures to note trends.

**[Tevatron Low-Beta Intersection Regions]{.underline}**

  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------
  **[Component]{.underline}**   **[No.]{.underline}**   **[Spares]{.underline}**   **[Risk]{.underline}**   **[Comment]{.underline}**

                                                                                                            

  PS (Magnetics)                10                      1                          H                        Had one failure. Long-lead
                                                                                                            time to replace. (See
                                                                                                            discussion below.)

                                                                                                            

  Q2 with 6-Lead Box            4                       1                          M                        Shielding makes it difficult
                                                                                                            to reach (See discussion
                                                                                                            below.)

                                                                                                            

  Low-Beta Quads                22                      8                          L                        Six different types. A single
                                                                                                            complete spare plus one spare
                                                                                                            cold mass exists for each
                                                                                                            type. (See the section on the
                                                                                                            TD/BD Study on Magnet Spares.)

                                                                                                            
  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------

[Low-Beta Power Supplies (Magnetics)]{.underline} have suffered one
failure in recent years. The component is a transformer that is a
special fabrication requiring about two months to replace. At present
there is a spare supply, but there is a risk of a double failure that
would shutdown operations.

*Recommendation*: Given the long-lead time, begin the procurement of an
additional low-beta power supply (\~\$30k) to be used as a spare.

[Q2 with 6-Lead Box]{.underline} replacement is made difficult due to
the concrete shielding that must be removed to gain access. Previous
estimates were that two months of rigging might be needed to free a box,
make the necessary change out, and restack the concrete. The situation
has since been exacerbated by the addition of more iron shielding.
However, it is felt that with a concerted effort the replacement might
be accomplished in much less time. Therefore, at this time it is not
known if a Q2 failure would cause a three-month downtime.

*Recommendation*: An attempt should be made to understand the time
needed to accomplish a low-beta Q2 and associated 6-lead box
replacement. If there is a potential for a long downtime, means to
reduce this time should be investigated.

**[Tevatron Cryogenics]{.underline}**

  ----------------------------- ---- ----------------------- --- -------------------------- --- ------------------------ ------------- ------------------------------
  **[Component]{.underline}**        **[No.]{.underline}**       **[Spares]{.underline}**       **[Risk]{.underline}**                 **[Comment]{.underline}**

                                                                                                                                       

  Centrifugal Cold Compressors  24                           3                              H                            Not           
                                                                                                                         sufficient    
                                                                                                                         spare parts   
                                                                                                                         on hand to    
                                                                                                                         ensure        
                                                                                                                         continued     
                                                                                                                         operations    
                                                                                                                         through the   
                                                                                                                         entire Run II 
                                                                                                                         program (See  
                                                                                                                         discussion    
                                                                                                                         below.).      

  Reciprocating Cold                                         4                              L                            These         
  Compressors                                                                                                            constitute    
                                                                                                                         additional    
                                                                                                                         spares to the 
                                                                                                                         Centrifugal   
                                                                                                                         Cold          
                                                                                                                         Compressors   

                                                                                                                                       

  Transfer Lines From CHL                                                                   H                            Exposed       
                                                                                                                         piping under  
                                                                                                                         busiest       
                                                                                                                         roadway on    
                                                                                                                         Laboratory    
                                                                                                                         site (See     
                                                                                                                         discussion    
                                                                                                                         below.)       

                                                                                                                                       

  Controls System                                                                           L                            Aging, but    
                                                                                                                         being         
                                                                                                                         addressed     

                                                                                                                                       

  Water Contamination in the                                                                H                            Cause could   
  Helium System                                                                                                          be human      
                                                                                                                         error or      
                                                                                                                         accident that 
                                                                                                                         opens system  
                                                                                                                         to damp air   
                                                                                                                         (See          
                                                                                                                         discussion    
                                                                                                                         below.).      

                                                                                                                                       
  ----------------------------- ---- ----------------------- --- -------------------------- --- ------------------------ ------------- ------------------------------

[Cold Compressors]{.underline} are custom-made devices with long-lead
times for replacement. The philosophy has been to keep on hand
sufficient replacement parts for those components most likely to fail,
such as bearings, rotors and impellers. The estimated life expectancy of
the installed components, and including the replacement components on
hand, still does not cover the anticipated duration of Run II.
Therefore, it will be necessary to augment the spare parts, the most
expensive and longest lead-time of which are the rotors.

*Recommendation*: Funding resources of about \$100k should be made
available to buy the needed spare cryogenic refrigerator components to
ensure the successful completion of Run II.

[Transfer Lines From CHL]{.underline} to the Tevatron are exposed as
they cross under the busiest roadway on the Laboratory site. Though
protected by guard rails and other barriers from a direct vehicular
accident, the piping could be damaged by falling debris, such as a gas
bottle, etc. The major concern is severing the liquid nitrogen lines
that might send the Tevatron through a thermal cycle. In that case,
recovery could require up to six months. In such circumstances, the
machine might be kept cold by trucking in liquid nitrogen and pumping it
in through one of the satellite areas. All of this would incur large
expenditures during the time of repairs, and of course, the Tevatron
could not operate until the transfer lines were repaired.

*Recommendation*: Install a culvert or some other cover that would
protect the exposed CHL piping to the Tevatron from a potential damage
from an accident.

[Water Contamination in Helium System]{.underline} and subsequent
clogging of the refrigerators could result from \"human error.\" In this
case the system would have to be warmed and the contamination eliminated
by purifying and conditioning with warm nitrogen and helium. The length
of time for recovery would depend on the extent of the contamination,
but it might amount to many months. Furthermore, prematurely starting
operations without adequate conditioning would just result in a
continual series of clogged filters.

A near miss occurred about one year ago when a spool piece was replaced
and a water cooling line was interchanged with a helium gas line. This
dumped water into the crygenic system requiring the removal and
replacement of ten Tevatron components. Operations were recovered within
a week, however a more serious disaster was avoided only because the
error was identified quickly enough such that only a few elements were
contaminated. The erroneous hookup involved a long-time employee of the
Laboratory, however, he was working in an unfamiliar environment. To
alleviate such problems in the future, the fittings on the pipe ends
will no longer be the same for the Helium and for the water pipes making
such an interchange impossible, however, this change out has just begun
and probably will continue for some time.

*Recommendation*: In the case of the change out of the Tevatron spool
piece water pipe fittings, the work should proceed as planned. However,
it would be advisable to keep a record of the devices where the change
has yet to be completed to be sure the change out occurs in a timely
manner.

**[Global Systems]{.underline}**

  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------
  **[Component]{.underline}**   **[No.]{.underline}**   **[Spares]{.underline}**   **[Risk]{.underline}**   **[Comment]{.underline}**

                                                                                                            

  Controls System Computers     30                      35                         L                        Uses out-dated VAX computers
                                                                                                            (See discussion below.)

                                                                                                            

  Electronic Modules and Their                                                     M                        In the case of a fire or other
  Spares Stored in Same                                                                                     mishap the backup device might
  Location                                                                                                  be lost (See discussion below)

                                                                                                            
  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------

[Controls System Computers]{.underline} are currently VAXes that are no
longer manufactured or supported commercially. At present, it is still
possible to obtain tens of VAX computers on the used market in a short
time, however, this situation might change in the future. The controls
system is being migrated to other machines, but the completed change
over will require two or more years.

The risk of a long downtime would be due to fire, lightning, or some
other mishap affecting the Beams Division computer room in such a way
that over half of the VAX inventory were damaged. Even then, with the
current market in used VAX computers, and with a concerted effort, it is
believed that the Controls System could be made operational in less than
three months.

[Electronic Modules and Their Spares Stored in Same
Location]{.underline} could pose a risk in case of an accident or a fire
in which everything was destroyed. This is a question of procedure and
the extent of the practice is not known. Therefore, assessing the amount
of risk is not easy.

*Recommendation*: Procedural questions were not considered to be part of
this study, however, the practice of storing spares with the original
equipment might be widespread. Even though there are no instances in
which a disaster resulted from this practice, the situation warrants
more investigated to determine the amount of risk involved.

**[Site Infrastructure]{.underline}**

  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------
  **[Component]{.underline}**   **[No.]{.underline}**   **[Spares]{.underline}**   **[Risk]{.underline}**   **[Comment]{.underline}**

  345-KV Switchgear KRS         1                       0                          H                        (See discussion below on the
                                                                                                            Power Grid.)

  345-KV Switchgear MSS         1                       0                          H                        30 yrs. old (See discussion
                                                                                                            below on the Power Grid.)

  345-KV Transformers           2                       0                          M                        With present grid
                                                                                                            configuration if one
                                                                                                            transformer were lost the
                                                                                                            physics program could continue
                                                                                                            at a reduced level (See
                                                                                                            discussion below on the Power
                                                                                                            Grid.)

                                                                                                            

  Central Utility Building                                                         L                        Not considered to be a risk at
  Chillers                                                                                                  this time (See below.)

  Cooling Ponds                                                                    L                        Silting has occurred in
                                                                                                            Tevatron ponds around the
                                                                                                            ring, but not considered to be
                                                                                                            a risk at this time (See
                                                                                                            below.)

  345-KV Power Poles                                                               L                        Wooden poles are attacked by
                                                                                                            birds, etc, but not considered
                                                                                                            to be a risk at this time (See
                                                                                                            below.)

                                                                                                            
  ----------------------------- ----------------------- -------------------------- ------------------------ ------------------------------

[Power Grid]{.underline} critical components include the 345-KV power
transformers and related hardware at the Kautz Road Substation (KRS) and
the Master Substation (MSS). There was a failure of the transformer at
the MSS in 1985 that caused a great upheaval in the physics program for
the better part of one year. The failure was investigated \[1\] and
various modifications were identified to make the site less vulnerable.
These included a more robust design for the transformer and the addition
of the KRS to provide a level of redundancy. This also allowed the load
to be divided between the two substations thus prolonging the life of
the individual components. If Fermilab were to suffer a similar loss of
a transformer as it did in 1985, it is believed that the power grid
could be reconfigured to retain operation, but at a reduced level, e.
g., the Main Injector repetition rate would have to be reduced. A
replacement transformer would cost in the \$1.2M range.

The review of the MSS transformer failure identified a number of
potential sources, though the most probable source was later identified
to be harmonic oscillations on the Fermilab side of the power grid.
During the investigation of related accelerator power supply failures,
oscillations as large as 30% were observed on the grid. It is believed
these caused premature aging of the power transformer insulation.
Subsequently, harmonic filters were added to reduce the harmonics to the
industrially accepted standard of less than 5% \[13\]. These filters are
now critical elements in maintaining the integrity of the power grid.
There are four resistors costing about \$10k each, two are used in the
Main Injector and two in the Tevatron. There is currently one spare for
each machine.

*Recommendation*: An additional spare resistor (\~\$10k) should be
purchased for each of the harmonic filters for the Main Injector and
Tevatron.

Finally, there are two sets of 345-KV switchgear, one each on KRS and
MSS. The one on MSS is thirty years old, and uses an outdated oil-based
design. If there were a failure of one of these components it would be
as catastrophic as losing a 345-KV power transformer. The responsibility
for the switchgear rests with Fermilab, and there is no spare for
either. The estimated replacement cost is \$200k.

*Recommendation*: The best situation would be to purchase one spare
345-KV switchgear (\~\$200k). The Laboratory should at least investigate
the availability and time to obtain a replacement device from Com Ed, or
elsewhere, in case a failure occurs.

The remaining three items in the infrastructure: chillers, cooling
ponds, and power poles; are under constant review and the belief is that
none poses a three-month downtime threat to operations \[14\]. The
backup plans for each of these items are discussed below.

[CUB Chillers]{.underline} have been upgraded with five new chillers and
three new heat exchangers (plate and frame) using Utilities Investment
Program (UIP) funds. The new system allows isolation of comfort-chilled
water from the process-chilled water system greatly reducing failure
rates. As a backup, there is a cross connection option that permits
connecting one comfort chiller to the process-chilled water system if
needed. Along with this, there are outside quick connects that permit
connection to a rental chiller unit to either chilled water system in an
emergency. With all of this in place, it is believed there is sufficient
capacity to supply the chilled water needs of the Collider program.

[Cooling Ponds]{.underline} have long been allowed to deteriorate. For
instance, the Tevatron ring cooling ponds are about one-third of their
designed effectiveness. Dredging is considered to be expensive
(\~\$1-2M). However, the degeneration of the Tevatron ponds is not a
concern at this time because the cooling load has been reduced by 50% or
more with the removal of the Main Ring and its replacement by the Main
Injector. The residual cooling loads are the Mycom compressors for the
Tevatron, electronics, and a few standard magnets that were not moved to
the Main Injector. Each year a diver checks the intakes for the pumps,
and generally reports increased silting around the intakes. However, the
opinion continues to be that the pumps will continue to pump water. This
spring, FESS plans to implement a complete clean-out of one pond on the
site and from that experience better understand how much effort is
needed to increase the reliability of the pond cooling system. In the
meantime, if a failure should occur, it is believed that a pump
replacement, or the effort to un-silt an intake could be accomplished in
a short time. There is also the fallback position of installing a
temporary pump and generator in place of a failed unit.

[345-KV Power Poles]{.underline} are inspected every 5 years, with the
next inspection to be conducted next spring. In addition, each year
woodpecker holes are patched with epoxy. Currently, there are three
spare poles in outside storage, any of which can be installed in a short
time to replace a fallen pole.

Replacement of all of the 345-KV power poles has been included on the
list of General Plant Projects (GPP). The project has been identified as
a need, but no year has been set for the replacement. A better
understanding of the requirement date will be known after the report on
the condition of the poles is received this spring.

**[Technical and Beams Division Study on Magnet Spares and Other
Issues]{.underline}**

Technical Division (TD) is responsible for all superconducting magnets
and spools, as well as all conventional magnets, except for kicker
magnets and a few spares in the proton source. The industrial facilities
to fabricate, repair and store these large devices also exist in the
Technical Division. The question of a potential three-month downtime due
to the lack of spares or tooling was posed to Technical Division
personnel. A summary of the current status appears in various review
presentations \[15\]. In summary, the Collider physics program is not in
immediate jeopardy of long downtimes given the number of spares on hand
and the magnet failure rates observed in recent years.

A spate of simultaneous losses, or the loss of an entire string of
magnets, can tax the capability of the TD facilities. An example of this
occurred for the \"infant mortality\" in the Main Injector quadrupole
magnets. Since 1998, six out of the 130 installed IQB magnets have
failed, leaving twelve spares of varying quality. Two out of 32
installed IQC magnets and three out of the 48 IQD magnets have failed or
been damaged beyond easy repair. Design faults in the original potting
of the magnet insulation have been identified and are being corrected as
magnets are repaired. At this time, the pipeline of spares undergoing
remedial action has been able to keep up with demand. For instance, all
IQC spares have been rebuilt, and one new IQC and a IQD have been
fabricated. However, the pipeline must continue in operation for the
foreseeable future. It is planned to fabricate four additional each of
the IQC and IQD magnets \[16\].

It should be mentioned that there are two quadrupole magnets in the Main
Injector for which there are no spares, the IQHs. However, in the case
of a failure an IQE can be modified into an IQH in a few hours, so there
are no magnets in the Main Injector system that do not have a spare.

In the Tevatron, the \"infant mortality\" occurred during the first
decade of operations. It has subsided over the last decade as design
faults were corrected, the last of these being the breaking of nylon
ties allowing the current buses to rub together and short. This has
reduced the number of dipoles removed from the Tevatron per year to
about three, out of which only about one dipole per year is lost
permanently. Therefore, it is believed there are adequate spares (\~17
dipole spares of each type) in the superconducting magnet inventory to
keep the machine operational for many years.

Losses of large numbers of superconducting magnets have traditionally
occurred during machine warm ups and cool downs, for instance in 1989
there was a loss of 33 magnets. Such a large loss did not occur in the
latest warm up in 1997, or the subsequent cool down for Run II.
Nevertheless, maintaining a program to resurrect additional spare
magnets from those that have previously been removed from the machine
would be wise. The fabrication of new magnets of the Tevatron design
would be very difficult, if not impossible, due to the lack of cable and
the long-lead time needed to commission the \"moth balled\" tooling.

The handling and specification of spare magnets has been somewhat
problematic, though any components can probably be located after a few
days\' concerted effort. Use of \"bar code\"or \"rf\" identification,
coupled with GPS location finding, could improve the data base
integrity.

Tevatron specialty magnets, such as the low-beta quads, and their
associated spool pieces might present a degree of vulnerability. There
is at least one spare magnet for each type of low beta quadrupole, plus
an additional cold mass. In an \"human error\" incident a few years ago
the CDF detector was opened and compressed a set of low-beta quads.
Ultimately, none of the magnets were lost, however, the accident
resulted in each low-beta quadrupole on one side of the interaction
region requiring a cold test at the Magnet Test Facility to confirm its
performance. To free the components, large amounts of concrete shielding
had to be removed from the tunnel. Though this required many months of
effort, it did not delay the Collider physics program as the Tevatron
was not yet ready for operations at that time. A similar event now might
result in a shut down of operations beyond the three-month downtime
limit as well as leave the system vulnerable to a single element failure
after the single spares were installed.

With the recent addition of rebuilt B and D spools, there is an adequate
number of Tevatron spare spool pieces on hand to ensure many years of
operations. Furthermore, the usual failure mode is the loss of the
corrector elements within the spool. For this type of failure, even in a
situation when all of the spares were exhausted, it is believed that the
Tevatron could remain operational. With the exception of the trim
dipoles that steer the beam locally, the higher-order corrector magnets
affect the beam in a combined sense. The loss of one set of these
correctors can be compensated by increasing the strengths of the
remaining correctors. It might also require the interchange the spools
in regions where the Tevatron is more sensitive.

Other problems have taken spool pieces out of service; for instance a
magnet was dropped on a D-spool in the tunnel, a set of safety leads
were burned up, and various vacuum leaks have occurred. In general,
there are spools in storage that are deemed repairable and could augment
the spares inventory if the number drops below a safe level. However,
this can only be done over a long period of time and requires an
on-going remedial program.

One additional spool piece concern is the seven installed H-spools that
carry the magnet power leads. Reliability of these elements has been
good over the years, though the recent introduction of one spool with a
set of high-temperature superconducting power leads has yet to
demonstrate its long-term integrity. If it fails there is a spare spool
piece that has the standard power leads. However, when other H-spools
with the new power leads are introduced into the machine, they might
present an overall vulnerability if additional failures were
encountered.

*Recommendations*:

(1) The program to recover superconducting dipoles and spool pieces from
    those needing repair should be continued. The level of effort should
    be reviewed to see that it satisfies the future needs of the
    Collider program.

```{=html}
<!-- -->
```
(2) The use of \"bar code\"or \"rf\" identification, coupled with GPS
    location finding, should be implemented throughout the accelerator
    complex to improve the data base integrity for installed and stored
    devices.

**[Related References, Reports and Memos]{.underline}**

1.  \"Report on Investigation of the Transformer,\" Larry Coulson,
    Chairman, FNAL Preliminary Report Issued March 8, 1985.

2.  \"Tevatron Magnet Spares Requirements,\" John Marriner, Internal
    FNAL Memo, July 26, 1995.

3.  Memo from Ray Hanft, June 28, 2002

4.  \"The proton Driver Design Study,\" Fermilab-TM-2136 (December,
    2000).

5.  Private communication from Frank Brumwell, IPNS, Argonne National
    Laboratory, 2002.

6.  Private communication from Dick Werbeck, LANSCE, Los Alamos National
    Laboratory, 2002.

7.  Memo from Bob Webber, August 14, 2002.

8.  \"Introduction to Reliability Theory,\" B. Schorr, CERN Report
    74-16.

9.  \"Fermilab Linac Status and Outlook,\" Talk presented by Bob Webber
    to the Laboratory Directorate, August 9, 2002.

10. \"Analysis of the Lifetime Data for the Linac 201-MHz Power
    Amplifiers,\" E. McCrory and R. Webber, Fermilab-TM-2178 (July,
    2002).

11. \"Trip Report to Burle Industries,\" E. McCrory (June 24, 2002).

12. B. Brown et al., \"Time Dependence and Temperature Stability of the
    Permanent Magnets for the Fermilab Antiproton Recycler Ring,\" IEEE
    Trans. On Appl. Superconductivity, **10** (March, 2000) 879.

13. Private communication from Dan Wolfe, Fermilab, 2002.

14. Private communication from David Nevin, Fermilab, 2002.

15. \"Superconducting Magnet Support for the Tevatron in Run II,\" Talk
    presented by David Harding at the Fermilab SC Magnet Internal
    Review, April 25, 2002.

16. Private communication from David Harding, Fermilab, 2002.