Maintenance atJefferson Lab
Jacque H. Ludwig

Workshop on Accelerator Operation
GUAS, Hayama and KEK, Tsukuba, JAPANMarch 10-14, 2003

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Introduction
The JLAB Operability Group is headed by Steve Suhring,
I am Jack Ludwig, Group Deputy.

The Operability Group is responsible for:••••

Maintenance oversight of the accelerator
Machine improvements
Shutdowns
Repairs

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Continuous Electron Beam Accelerator Facility
0.6 GeV linac(20 cryomodules)
1497 MHz67 MeV injector(2 1/4 cryomodules)
1497 MHz
Gain switched
diode lasers
499 MHz,∆φ = 120°AB

RF separators
499 MHzCB

A
B
Pockels cell

CA

GunC

Double sided
septumThomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

CEBAF Specifications
Polarized electrons with up to ~80% polarizationCurrent from 180uA to 10pA
Design Energy: 4GeV
Operating Energy: up to 6.0GeVEnergy Spread ~10-5
Simultaneous beam delivery to three experimental areas.
24 hours per day operation. On-Call Support by ~60 people

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

CEBAF Major Components• Tunnel ~ 1.4 km long, 10 m underground• Electrical Distribution – 40 MW• Low Conductivity Water ~ 400,000 liters at 1MΩ• Beamlines 7 km at 10-6 torr – 10-13 torr• 2 Polarized Photocathode Electron Guns
• 42 Cryomodules with eight 5 cell Superconducting
Niobium Cavities each
• 2 Kelvin Helium Refrigeration Plant• 80,000 liquid liters at ~ 210 g/sec

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

CEBAF Major Components (cont.)• 338 5kW klystrons each with independent controls• ~ 2100 Magnets and Power Supplies• EPICS. Experimental Physics and Industrial ControlSystem• CAMAC and VME with ~ 65,000 I/O Control Points
250,000 EPICS Records 140 IOC’s, and 80 UNIX
Machines

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Accelerator Availability

100.0%
80.0%
60.0%

Physics

40.0%

Beam Studies
Overall

20.0%
0.0%
FY98

FY99

FY00

FY01

FY02

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Availability Goal FY03 Q1Accelerator Availability Goal FY03
90%85%G0

HAPPEX

80%Weighted Average
Availability = 71.6%
75%

70%

8/29/03

8/3/03

7/10/03

6/29/03

5/28/03

5/17/03

5/8/03

4/27/03

4/25/03

4/17/03

4/2/03

2/3/03

1/23/03

12/19/02

12/5/02

11/18/02

11/16/02

10/28/02

10/26/02

10/18/02

10/16/02

60%

10/1/02

65%

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Accelerator Utilization
50.0
40.0
30.0

Maintenance

20.0

Shutdowns andHolidays
Beam Operations

10.0

FY02

FY01

FY00

FY99

FY980.0

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Utilization – Scheduled vs Actual

Scheduled (Weeks)

FY98

FY99

FY00

FY01

FY02

Average

Maintenance

2.9

2.0

2.6

3.7

0.8

2.4

Shutdowns and Holidays

7.4

11.0

11.3

10.7

17.9

11.7

Beam Operations

41.9

39.1

38.4

37.7

33.5

38.1

Actual (Weeks)

FY98

FY99

FY00

FY01

FY02

Average

Maintenance

2.9

2.1

2.3

4.6

1.1

2.6

Shutdowns and Holidays

7.8

11.39.5

10.6

15.3

10.9

Beam Operations

41.5

38.8

40.5

37.0

35.8

38.7

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Maintenance• Making repairs after a failure• Periodic replacement of high-wear parts• Correcting inspection deficiencies• Post repair testing• Calibration• Alignment• Equipment and software upgrades

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Safety Guidelines for Maintenance
Most maintenance tasks are performed within the guidelines
established by the Jefferson Lab EH&S Manual, Section
3000, Planning for Safe Operations.
Work control documents associated with these tasks include:• Standard Operating Procedures (SOPs),• Fire Hazard Work Permits,• Confined Space Entry Work Permits,• Electrical Service Work Permits, and• Radiological Work Permits.

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Operability Group Responsibilities:• Identify immediate maintenance repairs• Identify major causes of downtime• Conduct the weekly Maintenance Meeting• Assist in planning and scheduling maintenance days
• Apprise Program Deputy of proposed maintenancetasks and potential impact on machine operability.
• Collect, distribute, and track repair escalation reports.
• Oversee the safety and workmanship of maintenanceactivities in conjunction with the Site Safety Warden.
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

CEBAF Shift Log

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

12 Month Roll Up 12/00-11/01
TJNAF 12 Month Lost Time Totals by System
Lost Time (Hours)

480
Injector Vacuum Event

432
384

Fast Shut Down Trips

336

Magnet Cooling Problems

288

RF Separator Amplifier Failures, Control Module Communication Problems

240
192

Pipe Failure & Repair

144
96
48

SRF
0.2%

MPS
0.2%

RAD
0.3%

Ops
0.4%

PSS
0.5%

Control N
et 0.5%

Diag0.7%

Ops
ST 1.2%
Vacuum0.9%

Cryo1.2%

Other 1.5%

Sft 1.5%

Plant 2.7%
Ops
UT 1.9%RF 4.3%

Mag
5.3%

FSDTrips 5.3%

Guns 7.7%0Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Major Sources of Lost Time in 2002
• Machine Set Up and Tuning• Fast Shut Down Interruptions• Injector Instability• Vacuum Events• Site Power Outages

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Maintenance Categories• Scheduled tasks are planned in advance and approvedby Operability Manager.
• Standby tasks are approved and could be accomplishedopportunistically in the event of an unscheduled down.
• Unscheduled maintenance consists of repairs to correct
conditions that impede the scheduled program or present
safety hazards.
• Preventative maintenance is generally performed during
our Scheduled Accelerator Down (SAD) periods.

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Scheduled Maintenance• Weekly Maintenance Scheduling Meeting• Maintenance Task List
• SAD or Down Calendar

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Standby MaintenanceApproved tasks which could be performed opportunisticallyin the event of an unscheduled down, compiled from:• Maintenance Task List
• OPS-PRs• Daily 0800 Status Meeting• Crew Chiefs Shift Logs

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Unscheduled Repairs• Operators are the first responders to any malfunction
impacting beam delivery or beam quality.
• Identify – describe symptoms, isolate problem• Document – Log entry or OPS Problem Report (OPS-PR)• Under the direction of the Crew Chief, Operators initiate
corrective action – reset, reboot, make repairs, or contact
available on duty maintenance or on-call support.

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Lab Support Groups• Electrical Engineering Support (EES)

Repair Response• Accelerator Engineering Department

Cryogenics, Installation,
Magnets & Power Supplies, HighPower RF• Center for Advanced Study ofAccelerators (Accelerator Physics Group)

Diagnostics, Beam Dynamics• Institute for Superconducting RFCryomodules, Low Level RF• Plant EngineeringWater, Power• Polarized Gun GroupElectron Source Redundancy• Radiation Control Group

Radiation Control• Safety Systems Group

Personnel Safety and MachineProtection• Software Group

Applications, Alarms, Archivers
Thomas Jefferson National Accelerator Facility

Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Major Unscheduled Repairs• Crew Chief notifies the Program Deputy immediately ifinterruption is anticipated to be longer than one hour.
• Crew Chief notifies the Operability Manager immediately
if interruption is anticipated to be longer than four hours.
• Crew Chief determines when on-call help is required, andgenerates a Downtime Log and OPS-PR entry.
• If the problem is not solved two hours after the original
program interruption, Crew Chief initiates the repair
escalation process.

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Repair Escalation
Two hours after the original interruption, Crew Chief contacts:• the responsible maintenance Group Leader• the Systems Integrator• the Program Deputy• The Crew Chief describes the problem, reports present repair
work status, and may suggest a course of action.

• Escalation contacts may provide additional resources, orThomas Jefferson National Accelerator Facility
undertake an alternate
course of action.

Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Preventive Maintenance at CEBAF• Regular replacement of water and air filters• Regular replacement of belts, hoses, and lubricants• IR detection of electrical connections: 3 year cycle• Cleaning of HV cables and components every 6 months• Vibration / frequency analysis of rotating equipment every 3months• Citric acid flushing of magnets and power supplies

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Accelerator Work Planning

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig

Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy

WAO 2003, March 10-14, 2003, Jacque H. Ludwig