FERMI NATIONAL ACCELERATOR LABORATORYACCELERATOR MAINTENANCE
Mau. R, FNAL, Batavia, IL 60510, USA
Fermilab is a U.S Department of Energy (DOE) research laboratory, operated under DOE contract by Universities Research Association (URA).

Abstract
Fermilab has two-hour necessary repair periods, 8 to 12
hour shutdowns, and multi-week shutdowns. The problemof any shutdown is how to communicate the goals andplans, how to coordinate the work, and how to ensure that
only approved work gets done. This paper addresses theaspects of coordinating, communicating, planning, andcontrolling shutdowns, and the role Operations plays inthese activities.

RUN IT TILL SHE BLOWSThis is the philosophy of accelerator operation atFermilab. In general we do not schedule repetitive
shutdowns. Instead, we wait until something triggers ashutdown.
Here are a few examples of triggers:• Equipment failures• When the broken equipment list becomes large
enough• Necessary upgrades• The experiments need an accessIn order to make this system work, the Operations
Department must keep an up-to-date set of work-lists in
preparation for any triggered shutdown.

MAINTENANCE SCHEDULES
One of Fermilab’s greatest strengths is that of being
extremely flexible. A fixed schedule is never allowed to
get in the way of doing the right thing.
However, Fermilab’s greatest weakness is that of being
extremely flexible. Flexibility opens schedules up topressures, whining, and whims. Everyone has an opinion
and thinks their plan is the only one that is correct.
Trying to balance out these two opposites requires
knowledgeable decision makers who possess the strengthof will to stand by their decisions.

SCHEDULE CONTROLSThe problem with controlling this type of maintenancesystem boils down to ensuring that people, who are justtrying to be responsible about their systems, don’t get
carried away. How does it work?
• First, our people are very responsible. Theyuse every opportunity to check and adjust theirsystems so that they cause no down time.
However, during a shutdown they want to do
too much. So we like to remind everyone that,“If it ain’t broke, don’t fix it!”

In Fermilab’s realm of schedules,
One Size Does Not Fit All.

TYPES OF ACCESS
Fermilab has two types of access.

Controlled Access•••

The tunnel interlocks have not been dropped.
The tunnel requires no search and secure.
Usually, this type of access has limits imposed on
the amount of personnel and time of access.

Supervised Access•••

The tunnel interlock system has been dropped.
Operators must search and secure the area.
Radiation technicians must perform a radiation
survey of the tunnel. This means that beam mustbe turned off for several hours before performing
the survey.
An example of the downtime required for an eighthour shut down would be shutting the beam off at0500 hours (5 AM) and starting to turn things backon at 2000 hours (8 PM). This is a total of fifteen
hours down for an eight-hour shutdown.

COORDINATION MEETINGS
Keeping the castle walls from being stolen requires
good communication.
• The Run Coordinator holds 0900 hours meetings
every Monday, Wednesday, and Friday. The Crew
Chief, experiment representatives, and Machine
Coordinators report on accelerator operations.
• The Run Coordinator holds 1030 hours study
meeting. This meeting is attended by MachineCoordinators and other associated people.
• Impromptu meetings of people bumping into eachother are always being held in tunnels, collisionhalls, or parking lots.
• Both the Crew Chief and the Run Coordinator haveregular phone meeting with each other, with
Machine Coordinators, and with the people doing
the work.
• The Environmental, Safety, and Health group
review all proposed work.

TYPES OF SHUTDOWNS
Equipment Failures
Here is a list of things that Fermilab keeps in mind
during an equipment failure.
1. Trigger Examples• An unknown failure occurs. Send experts toinvestigate and wait for an estimate before
doing anything else.
• If the failure is known, such as a Tevatron
high-field quench, then there will be aminimum downtime of one hour. Then thequestion arises, can anything be accomplishedduring the recovery, like?
a. Fix other problems
b. Conduct accelerator studies
c. Do nothing
2. Decision Makers• Crew Chief• Run Coordinator• Operations Management (sometimes)
3. Pressure• Necessary Repair Lists• Machine Coordinators• Department Heads• Key Support Department Personnel
4. Tricks
2. If there is some special trick that can fix thisproblem, don’t spread the word until you haveenough information to make a decision.
5. Access Type
3. Try to keep the access controlled.

Short Shutdowns
What if the problem only requires a short shutdown of
two hours or less? Here’s how it may go.
1. Trigger Examples• A pump is making strange noises. Expertssuggest replacing it within the next twentyfour hours.
• In this situation we will look for “a window ofopportunity.”
2. Decision Makers

• Run Coordinator• Operations Management• ES&H (possibly)
3. Pressure• The 0900 hours meeting on Monday,
Wednesday, and Friday.
• What’s the potential for failure?
• Machine Coordinators• Department Heads
4. Tricks• Try scheduling a noon repair. (You willquickly discover who wants the work doneand when.)• The decision makers must limit the approved
work and not let things get carried away.
5. Access Type• Keep the access controlled.

•••
•
•
3.

4.

Four Hour Shutdowns
Here’s where things may start getting serious.
1. Trigger Examples• Two Booster power amplifiers are down.
• The experiments want a one-hour access.
• A magnet needs a vacuum leak check.
• Do we really need to go down? NO! However,if there is enough complaining going on wemay shut down.
2. Decision Makers• Run Coordinator• Operations Management• ES&H (possibly)
3. Pressure• The 0900 hours meeting on Monday,
Wednesday, and Friday.
• What’s the potential for failure?
• Machine Coordinators• Department Heads
4. Tricks• At all costs, try to schedule this type of accessfor the beginning of the next day.
• You need time to plan, and you need time torecover.
5. Access Type• This should be a controlled access, but theexperiments frequently ask for supervisedaccess.
• This adds time because operators must
conduct a search and secure before the start up
can begin.

One or Two Day Shutdowns
1.

Trigger Examples• The work list has grown large.
• A new collimator and shielding is ready for
installation in the Booster.
2. Decision Makers• Run Coordinator

5.

Division Head guidance
Program Planning guidance
Operations ManagementES&H
Discussion for this type of access and easily
begin up to one month ahead of time, or in
reality at least a week a head of time.
Pressure• The 0900 hours meeting on Monday,
Wednesday, and Friday.
• What’s the potential for failure?
• Machine Coordinators• Department Heads• Division HeadTricks• At all costs, try not to schedule this shutdown
just before a weekend.
• Schedule this work to begin at the beginningof the day shift.
• Schedule this shutdown to begin on aWednesday. It give you time to plan.
• Understand that the machine will require timeto recover.
Access Type• For a one-day access, try to keep the access
controlled, but there will be pressure for one
day of supervised access.
• If it’s a two day access, there will be evenmore pressure to go to a supervised access.
• A supervised access means that time must be
allotted for interlocking areas.

A Shutdown of One Week or Greater
1.

Trigger Examples• Budget Concerns• Long range upgrades• In the past we’ve made shutdown decisions
due to a coal strike and to nitrogen delivery
problems. (The coal strike probably needs alittle explanation. One particularly cold winterthere was a coal strike. This should not havebeen a problem to coal fired electricalgenerators, except that it was so cold that thepiles of surplus coal were frozen. TheDepartment of Energy called Fermilab andasked them to shut down.)
2. Decision Makers• Lab Director• Division Head• Run II strategy meeting• Run Coordinator• Operations Management
3. Pressure• Division Head and Lab Management• Department Heads• Much heated discussion
4. Tricks

••5.

Assign a shutdown coordinator.
This schedule is generally planned out severalmonths in advance and requires manymeetings.
• You need to plan.
• You need time understand how much time you
will need to recover.
Access Type• This is a supervised access.

Emergency Power Reduction Shutdowns orPower Outages
1.

Trigger Examples• There is a problem with the local power grid.
a. When there’s a problem with a local grid,
the power reduction usually occurs
between noon and 2000 hours.
b. The power company must give a 24-hourwarning.
c. There is a cost savings for the laboratory,
which is announced the day before.
d. The power company and the laboratory
discuss the amount of power reduction theday before it is to occur, such as reducing
our normal usage of 47 MW by 20 MW.
• A kite string has draped over a power line.
• Coal strike (see previous explanation)• Loss of a major power line insulator
2. Decision Makers• The Lab Director will make all decisions forthe laboratory.
• The normal decision makers must pick a
shutdown coordinator.
• The laboratory Power Coordinator (OperationsDepartment) will implement power reduction
policies working with a special power
reduction team.
3. Pressure• AT ALL COSTS KEEP THE CRYO
SYSTEM UP AND RUNNING.
4. The Shutdown Coordinator Responsibilities• Take advice from all the decision makers andthen pick a tentative date for the shutdown atleast six months ahead of time.
• Take advice and the pick a “set in stone date”
no later than one month ahead of time.
• Figure out the shutdown’s limiting factors.
a. Lengthb. Manpower* Is the manpower available?
* Outside lab temporary help?

*Draft people from other departments
and divisions?
c. Other Pressures
4. Meetings — Communication is Critical!
• The Division Head formally sets the tone forthe shutdown.
• Meet with Division Management so they canunderstand what is about to happen and set thescope of priorities.
• Department Head meetings• Informal weekly meetings with those directly
impacted by shutdown.
• The 0900 hours meeting on Monday,
Wednesday, and Friday.
• Special shutdown meeting where work lists
are analyzed.
• Run II strategy meetings• During the shutdown there are daily or every
other day meetings.
• ES&H meeting to review work lists

SUMMARY“Run it till she blows” works. It does give Fermilab
more up time.
System and support departments must be organizedahead of time in order to use down time whenever itappears.
There must be strong decision making.
The Run Coordinator administers all decisions
about beam operations.
A Shutdown Coordinator administers all decision
about long shutdowns.
A Power Outage Coordinator administers alldecisions about emergency power reductions.
The Accelerator Operations Department is heavily
involved in all these activities.

Fermilab is a U.S Department of Energy (DOE) researchlaboratory, operated under DOE contract by UniversitiesResearchAssociation(URA).