MAINTENANCE AT JEFFERSON LAB
J. Ludwig, Thomas Jefferson National Accelerator Facility
12000 Jefferson Avenue, Newport News, VA 23606, USA•

Abstract
Maintenance at Jefferson Lab is largely driven by theexperiment schedule: two or three major maintenanceand/or upgrade periods are generally scheduled each year.
Unscheduled maintenance involves critical repairs thatcannot be deferred to a Scheduled Accelerator Down
(SAD). Scheduled maintenance includes deferred andpreventative maintenance as well as planned modification
and upgrades to the accelerator.

1 INTRODUCTION
Maintenance refers to work performed on the hardware
or software of the Jefferson Lab accelerators to maintainand improve availability. Examples of accelerator
maintenance activities include:• Making repairs after a failure• Periodic replacement of high-wear parts• Fixing inspection deficiencies• Post repair testing• Calibration• Alignment• Equipment and software upgrades

2 RESPONSIBILITIES

••
•
•
•

•
•
•

Follow up on action items initiated at the Daily
Summary Meeting.
Assist in planning and scheduling maintenancedays, scheduled long-term off times, andaccelerator hot checkout and start-up activities.
Conduct the weekly Maintenance Meeting, which
is used to plan, schedule, and coordinate upcoming
maintenance and installation activities.
Participate in the distribution, collection, andreview of system checklists for accelerator
readiness after long-term off times.
Apprise the Program Deputy of proposed
maintenance tasks and the potential impact onmachine operability.
Identify major causes of downtime, recommend
methods for improvement, review projectdescriptions for upgrades that will improveaccelerator availability, and track those projects toconclusion.
Collect, distribute, track repair escalation reports.
Oversee safety, workmanship, and documentationof maintenance activities in conjunction with theAccelerator Site Safety Warden.
Review and initial the weekly Walk-Through andtrack any noted discrepancies to closure.

2.1 Program Deputy

2.3 Accelerator Operators

Program Deputies are Jefferson Lab staff who areappointed to serve for two-week periods. During his or
her period of service, the Program Deputy is responsible
for the accelerator program for all shifts. The Program
Deputy communicates with all shifts on a daily basis, andadditionally:• Conducts the Daily Summary Meeting each
workday morning where the Crew Chief Shift
Summaries for the previous 24 hours (or weekend)
are presented, develops any necessary short-termschedule modifications, and assures that theappropriate laboratory resources are utilized asnecessary to keep the program on schedule.
• Conducts the Weekly Accelerator Planning
Meeting to define the short-term operatingschedule for the upcoming two-week period.

Accelerator operators are the first responders to any
malfunction impacting beam delivery. They will identify,
document, and initiate corrective action. They mayperform repair work as specified by approved
troubleshooting procedures and play an important role inidentifying when specific repair work is required, e.g.
Daily Activity Log, Operations Problem Reports (OPSPRs), Crew Chief Shift Logs, and Walk-ThroughChecklists.

2.2 Operability Manager
The Operability Manager heads the Operability Groupand is responsible for maintenance oversight of theaccelerator. The following are the general responsibilitiesof this group:• Identify immediate maintenance needs based on
the experiences of the twenty-four hours prior tothe Daily Summary (8:00 a.m.) Meeting.

2.4 Lab Support Groups
Maintenance activities are supported by the Jefferson
Lab system support groups, who perform the maintenancetasks:• Electrical Engineering Support (EES)(Repair Response)• Accelerator Engineering Department•(Cryogenics, Installation, Magnets, Power
Supplies, High Power RF)• Center for Advanced Study of Accelerators(Diagnostics, Beam Dynamics)•Institute for Superconducting RF(Cryomodules, Low Level RF)• Facilities Management(Water, Power)

••
•
•
•

Polarized Gun Group(Electron Source Redundancy)
Radiation Control Group(Radiation Control)
Safety Systems Group(Personnel Safety/Machine Protection)Software Group(Applications, Alarms, Archivers )Vacuum Group(Beam Transport and SRF Vacuum)

3 MAINTENANCE
Maintenance tasks can be divided into three categories:• Immediate• Scheduled• Standby

3.1 Immediate Maintenance
Immediate maintenance consists of repairs to correct
conditions that impede the scheduled program or areidentified as safety hazards.
Immediate maintenance repairs are requested by theCrew Chief at the time of the failure. Downtime entries
are made in the appropriate computer-based logs (e.g.
Daily Activity Log) and include as much of the following
information as possible:• When the problem was first recognized• A thorough description of the problem• When the problem was resolved• When beam was re-established•3.1.2 Repair Escalation
The Crew Chief can call for outside assistance any time
there is a problem with the accelerator. However, when aproblem cannot be solved by control room staff, on-site
maintenance personnel, and/or on-call help within twohours of the original program interruption, the repair
escalation process is initiated by the Crew Chief.
NOTE: The repair escalation process serves as amechanism to identify recurring problems, educate anddraw on the knowledge of the entire Jefferson Lab staff
and, ultimately, improve availability.
The following guidelines are used in the event of arepair escalation:• Two hours after the original program interruption,
the Crew Chief will notify the following
designated escalation contacts:• The responsible maintenance Group Leader ordesignee• The Systems Integrator• The Program Deputy(This is to occur even if on-call help has had only
a portion of the two hours to correct the problem.
The Crew Chief must record the details of therepair escalation in the Daily Activity Log.)•3.1.1 Repairs
Repairs are performed by on-duty maintenancepersonnel whenever possible, but other staff, including
EH&S and Radiation Control Group staff, may be called
in as needed to evaluate work requirements, provide
support, or expedite resolution of the problem.
When a problem occurs, standard practice is for theCrew Chief to immediately evaluate the situation and takeappropriate action using available resources, including oncall help. Before requesting on-call help the Crew Chief
will use his/her best judgment as to whether the problem
can be corrected by the control room staff using available
system knowledge and troubleshooting guides. If aproblem persists longer than two hours, the repair
escalation process is initiated.
The following guidelines are used whenever anaccelerator repair is necessary:• The Crew Chief notifies the Program Deputy
immediately if interruption to the program isanticipated to be longer than one hour.
• The Crew Chief determines when the problem
cannot be corrected by the control room staff andon-call help is required. The Crew Chief thencontacts the appropriate on-call group according tothe information in the on-line call-in lists.
• The Crew Chief generates a Downtime Log andOPS-PR entry.

If the problem is not solved two hours after theoriginal program interruption, the Crew Chief
initiates the repair escalation process.

•••

The Crew Chief notifies the contacts that there is aproblem, describes the problem, reports the present
repair work status, and may suggest a course ofaction.
The escalation contacts can decide to provideadditional resources, undertake an alternate courseof action, or continue on the present course. In
order to optimize its response and to make the bestuse of available personnel, a maintenance group
can develop its own internal escalation policy toassure that the appropriate people are notified in atimely manner.
Four hours after the original program interruption,
the Crew Chief will notify the appropriate
Department Head (or his/her Deputy), theOperability Manager, and the Program Deputy ofthe problem and the present repair work status.
The Crew Chief must record the details of thesecalls in the Daily Activity Log.
The Crew Chief notifies the Operability Manager
immediately if interruption to the program isanticipated to be longer than four hours.

3.1.2.1 Repair Escalation Report
The Repair Escalation Report (RER) is a written reportthat summarizes escalated repair incidents.
TheOperability Manager can call for an RER for any repair
activity that exceeds four hours.

3.2 Scheduled Maintenance
Scheduled maintenance consists of maintenanceactivities that are planned in advance and approved by theOperability Manager.
As each maintenance task is completed, the work is
documented in the Daily Activity Log, and a problem
report solution entered in the on-line problem reporting
system (OPS-PR) when appropriate.
Two or three major Scheduled Maintenance Downtimes(SAD’s) are scheduled each year according to theaccumulated maintenance and upgrade requirements of
the accelerator and the experimental halls.

3.3 Standby Maintenance
The Operability Manager compiles a list of pending
maintenance tasks. Two subsets of this list include
planned and preventative maintenance tasks that could be
performed opportunistically in the event of anunscheduled down, and deferred repairs for items notimpeding the scheduled program.
The Operability Manager, in consultation with theProgram Deputy, uses the Standby Maintenance Task List
to initiate work through the Crew Chief when there is anunscheduled accelerator down. The Program Deputy can,at any time, curtail or cancel tasks that may threaten thescheduled program.
Tasks performed during an unscheduled down period
include: correction of the problems(s) that caused theprogram interruption, and scheduled maintenance that canbe performed ahead of schedule.
Currently, 10 to 12 hours per week are reserved fordeferred maintenance. If there is not sufficient cause touse these hours for maintenance, they are scheduled andutilized for Beam Studies (BS) for the development of theaccelerator.

4 WORK PLANNING PROCESS
Operability Manager maintains a list of pending
maintenance and upgrade tasks. The primary sources forthese tasks are:• OPS-PRs• Crew Chief Shift Logs
• Daily 0800 Summary Meetings• Weekly Maintenance Scheduling Meetings• Preventative Maintenance Procedures• Direct requests to the Operability ManagerThe Operability Manager is the gatekeeper for all workperformed on the accelerator. As such he evaluates eachtask request for criticality, impact to accelerator systems,and schedule.
Task requests may be approved, disapproved, returned
for further review, scheduled, or placed on the Standby
Maintenance List.
The present work request system is presently being
defined, refined and will soon be implemented via an online Accelerator Work Request system.

6 REFERENCES[1] Accelerator Operations Directives, Thomas Jefferson
National Accelerator Facility.