How We Do Business at APS

C.Y. Yao, Glenn Decker, Roger Klaffky,
Greg Banks
The submitted manuscript has been createdby the University of Chicago as Operator of
Argonne National Laboratory (“Argonne”)
under Contract No. W-31-109-ENG-38 with
the U.S . Department of Energy. The U.S.
Government retains for itself, and others
acting on its behalf, a paid-up,
nonexclusive, irrevocable worldwide license
in said article to reproduce, prepare
derivative works, distribute copies to the
public, and perform publicly and display
display publicly, by or on behalf of theGovernment.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoThe APS Facility
•
•

•

•
•
•
•
•

A Third generation dedicated synchrotron radiation user facility
Storage ring:– Electron beam
– Energy: 7 GeV– Operation current: 100 mA
Injector:– 650 MeV linear accelerator– 450 MeV electron accumulator ring
– 7 GeV booster synchrotron
Commission completed in 1995
In operation since 1996
Scheduled user time: 5000 hours per year
Best availability run: 99.06% (Run 3 of 2002)
Best mean time between faults: 71.76 hours

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoAPS Statistics: FY Beam Availability
100.0%
99.0%
98.0%
97.0%
96.0%
95.0%
94.0%
93.0%
92.0%
91.0%
90.0%
FY199803/10/2003 WAO 2003,
Hayama/Tsukuba,Japan

FY1999FY2000FY2001

FY2002

FY2003 (Partial)

Operations Group
Advanced Photon Source
CY YaoAPS Operational Statistics: FY MTBF (Hour)
80.0

70.0

60.0

50.0

40.0

30.0

20.0

10.0

0.0
FY199803/10/2003 WAO 2003,
Hayama/Tsukuba,Japan

FY1999FY2000FY2001

FY2002

FY2003(Partial)

Operations Group
Advanced Photon Source
CY YaoAPS Operational Statistics: FY Scheduled User Hours
5100.05000.04900.04800.04700.04600.04500.04400.04300.04200.04100.0
FY199803/10/2003 WAO 2003,
Hayama/Tsukuba,Japan

FY1999FY2000FY2001

FY2002

FY2003(Partial)

Operations Group
Advanced Photon Source
CY YaoAdvanced Photon Source

Accelerator Operations(AOD)

Accelerator Systems(ASD)

Experimental Facilities(XFD)

MCR OPS

RF

Exp. Facilities Eng.

OPS Analysis

Electrical

Magnet Devices

Beam Diagnostics

Control

Beamline Operations

Exp. Floor OPS

Accel. Physics

Optics & Metrology

Information Systems

MechanicalHigh Res. Scat.

Conventional Faclt.

VacuumPolarization Studies

Building Mgr.

Survey & Align.

X-ray Microscopy

Beamline Ctl.

X-ray Phsics

Technical Support

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoAPS Operations Group

Group Leader

Group Secretary

Crew #1

Linac/LEUTL
Staff

Crew #2

PAR/Booster
Staff

Crew #3

Controls
Staff

Crew #4

Storage Ring
Staff

Crew #5

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATIONS TEAM
•
•
•
•
•

16 operators in five crews with 3 members per crew
Three shift per day
Crew shift rotates backward weekly
12-hour shift on weekends
One backup week on normal working hours for:– Training– Interaction with staff members and system groups– Participating in project– Other activities• One rest week after a full rotation• 84% of time on shift work
• On a 40-hour-per-week base, operators spend 84% time onshift work.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATIONS TEAM (cont.)••

On-shift crew’s primary
responsibilities:– Handling machine
operation– Shift event logging– Fault reporting– Tunnel safety handling– Assist with emergency
response
During a user run, the crews
also perform certain off-hour
repair tasks:– Magnet power supply
swapping– RF station switching– Rad. Monitor replacement

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

•

•

The technical systems groups
provide on-call support andperform repair
Members of OPS staff’s
responsibilities:– Training documentation– Work coordination– Technical assistance– On-call support– Work approval– Software and proceduredevelopment

Operations Group
Advanced Photon Source
CY YaoOPERATRIONS TEAM (cont.)• Overall mean and median years of experience: 2.7 years
and 4.1 years• Average years of experience for crew chiefs: 7.8 years• Operator turnover rate: 15% in recent years.
• Factors affect turnover rate:– Rotating shift work schedules– Lack of career advancement opportunities– General employment conditions• New hires typically have 2 years of college education or
several years of industrial experience.
• Biggest challenge for the group: Maintaining a well focused,consistently performing, and motivated operations team.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoTHE MAIN CONTROL ROOM
•

•
•

•
•

•

The MCR is the center of many machine related activities:– Machine operation– Machine studies– Shutdown work coordination– Tunnel safety and group lock out tag out
Size of the MCR: about 2000 ft2
Three control consoles:– Injector control– Storage ring control– Machine study and comfort displays
A total of 11 double-head Sun workstations
Several video monitors:– Beam profile display– Remote scope display
Three oscilloscopes for linac and booster waveform display

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY Yao03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY Yao03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoTHE CONTROL SYSTEM• The control system is a distributed control system• Hardware:– Sun workstations– VME-based IOCs
• Software:– Unix/Solaris operating system– EPICS/VxWorks– Medm screens and alarm handlers– High level programming: Tk/Tcl and SDDS

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATOR TRAINING• Three categories:– Agonne-specified trainings– General operations trainings– Accelerator operation trainings• Argonne-specified trainings are provides by ANL, which
include:– CPR training– Emergency response– Fire protection training– Electrical safety– Radiation safety– Hazard materiel– Computer security, etc.
03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATOR TRAINING• General operations trainings are provided by APS and OPS
Group:– Tunnel access and control– Group lockout/tagout– Building/Facilities Safety– Accelerator physics for operators– Conduct of operations, etc.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATOR TRAINING (cont.)• Accelerator operation trainings are provided by the OPS
Group.
• Training on each machine consists of required readings,
required classes, on-shift training, and qualification tests.
• Training documents and courses are developed by staff
members of operations and technical systems groups• Videotapes of the training courses are available for viewing

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoQUALIFICATION PROCESS• A new operator is required to complete qualification on all
four accelerators within two years• On-shift training and self-guided training are the primarymethods• Crew chiefs and qualified operators conduct on-shifttraining• Qualification tests include written test, walkthrough andconsole operation test, which is administered by the OPS
staff members• Re-qualification is required every two years after initialqualification• Average qualification time is 12.2 months from date of hire.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATIONS LOGBOOK• The operations logbook:– An on-line electronic logbook serves as acommunication tool
– A paper form for archival purpose• The crews enter logbook entries using Framemaker.
• A hardcopy is printed at the end of each shift• Online web pages are automatically generated from the
Framemaker file and updated periodically.
• A search tool is used to review and reconstruct past events.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY Yao03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATION MODES•••

Various storage ring operation modes are developed toserve various user requirements.
Current operation modes:
1. Top-up, low emittance (2.3 nm-Rad.), 1+1x22 singlets fillpattern
2. Top-up, low emittance, 1+ 7x8 hybrids fill pattern
3. Non-top-up, high emittance (7 nm-Rad.), 1+1x22 singlets
fill pattern
4. Non-top-up, low emittance, 324 bunch fill pattern
Refill is performed twice a day in non-top-up.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATION MODES (cont.)•••

Top-up is scheduled for 75% of user beam time.
Advantages of top-up:– Higher average photon flux
– Better stability of both storage ring and beamline– Achieve lower emittance and higher photon brightness
without beam lifetime concern.
Non-top-up modes are scheduled for 25% of user beamtime.
– For parasitic injector and LEUTL study– For operator training– For injector maintenance and improvement.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATION MODES (cont.)
•

•
•

•

High emittance lattice or multiple-bunch fill patterns areused in non-top-up injection mode in order to achieve
longer beam lifetime.
Low emittance lattice and singlets fill pattern is used for
top-up operations.
The bunch fill pattern typically has one BPM bunch for BPM
timing, which is followed by either singlets or multiple
normal bunch trains.
New lattices and fill patterns are still being developed toaccommodate increasingly diverse user demands.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoBUNCH FILL PATTERN

Singlets: 1+1x22

Hybrid: 1+7x8…
03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoOPERATIONS SCHEDULE• Operation schedule is developed by AOD with input fromboth the beamline users and APS groups.
• Overall schedule time: 5000 user hours per FY• Three runs per FY• A month long maintenance period between runs for:– New beamline installation– Machine and system upgrades– Maintenance• A week long machine startup/studies period at the
beginning of each run• A 48-hour and an 8-hour machine intervention periods inevery 2 weeks for routine maintenance and machinestudies
03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoRun 2002-03
Oct
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

Nov
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
16
16
17
17
18
18
19
19
20
20
21
21
22
22
23
23
24
24
25
25
26
26
27
27
28
28
29
1 → 2930 1 →
30
31

Run 2003-01

1
1
1
1
1
1
1
1
1

Dec
→
→
→
→
→
→
→

Jan
1
2
1 3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

Top-up User Operation in low emittance mode
Non top-up User Operation
Fill pattern is 23 singlets unlessotherwise indicated03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Feb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

3
3
3
3
3
3

1
2
3
4
5
6
7
8
9
10
11
3 → 12
13
→
14
→
15
→
16
→
17
→
18
19
20
21
22
23
24
25
26
27
28
29
30
31

Run 2003-02

Mar

Apr
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

1
1
1
1
1
4
4
4
4
4

SOM Periods
1 Top-Up
Hybrid Fill - (singlet)
2 Hybrid Fill - (triplet)
3 110 mA - Low E. singlets
4 324 Singlets - Low E. Fill

May
June
1
1 4 →
2
2 4 →
3
3
4
4
5
5
6
6
7
7
8
8
9
1 → 9
10
10
→
11
11
→
12
12
→
13
13
→
14
14
→
15
4 → 15
16
16
→
17
17
→
18
18
→
19
19
→
20
20
→
21
21
22
22
23
23
24
24
25
25
26
26
27
27
28
4 → 2829 4 →
29
30 4 →
30
31 4 →

July
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

1
2
3
4
5
6
7
4 → 8
9
4 →
10
4 →
11
4 →
12
4 →
13
4 →
14
4
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

Machine Studies
Maintenance
Shifts set aside for Studies/
Machine Intervention as Needed

Aug

1
1
1
1
1
4
4
4
4
4
4

1
2
3
4
5
6
7
8
9
10
11
12
1 → 13
14
→
15
→
16
→
17
→
18
→
4 → 19
20
→
21
→
22
→
23
→
24
→
25
→
26
27
28
29
30

Weekends
Lab Holidays

Operations Group
Advanced Photon Source
CY YaoSeptUSER INTERACTION•••

User information:– Storage ring beam status display– An hourly updated fill history page
– Emittance & RMS beam motion display– User operation statistics page– logbook entries– PA announcements
User requests are passed to on-shift crew via the EFO Group.
Operation performs:– User steering– Insertion device gap handling– If necessary, a direct contact between OPS staff and users isestablished to resolve a request.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoMAINTENANCE PROGRAM
•

•
•

•
•

It is hard to form a uniform maintenance strategy because of– budget constraints– Non-standard nature of many accelerator components
Our approach: combination of "run it until it breaks" andpreventive maintenance.
Back-up systems or spare parts for– Storage ring RF stations– Magnet power supplies– Control system equipment– Beam diagnostics system equipment
Off-line maintenance can be performed.
Preventive maintenance is performed on storage ring dipole,kicker, and septum supplies, which do not have spares

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoMAINTENANCE PROGRAM (cont.)• Periodic preventive maintenance is performed on standard
equipment, such as:– Mechanical parts: motors, fans, etc.
– Electrical equipment: switches, breakers, lighting
equipment, etc.
– We are considering• Development of trend-analysis-based failure prediction inunder consideration in order to facilitate preventive
maintenance.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoMAINTENANCE PROGRAM (cont.)• The OPS Group's role in maintenance:– Work approval– Work Scheduling– Work coordination– Record keeping• Technical systems groups are responsible for:– Submitting maintenance requests– Performing work
– Verification the functioning of equipment

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoSOFTWARE TOOLS••

Many software tools have been developed to assist the operators.
The tools have improved productivity and consistency.
– Work request system– RM&D system (repair, maintenance and development)– Downtime report tool
– Operation statistics tool
– OPS Group mail list server/archive– Automatic work station configuration tool
– Save/restore/compare tool for machine configuration– Procedure Execution Manager– Scope waveform archiving and configuration tool
– Data monitoring and review•

More on this subject in our poster

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoDOWNTIME STATISTICS• Downtime statistics are part of the performance monitoringand improvement mechanism.
• The APS management has set an overall goal of– 95% beam availability– 48 hours MTBF
• Every group is also assigned a downtime allowance and an
MTBF goal• Storage ring faults and downtimes are automatically logged.
• The OPS Group performs downtime analyses and posts
downtime statistics on a weekly basis• Disputed items are discussed, resolved, and corrected.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoListing of Statistics for Run1-2003 (Created Thu Mar 06 09:06:36 CST 2003)
Total Amount of User Time in this interval 729.04 Hours
User periods in this interval----------------------------01/29/2003 08:00 To 02/03/2003 16:00 128.00 Hours, Delivered Beam: 122.36 Hours,
02/04/2003 08:00 To 02/10/2003 08:00 144.00 Hours, Delivered Beam: 141.70 Hours,
02/12/2003 08:00 To 02/18/2003 08:00 144.00 Hours, Delivered Beam: 140.33 Hours,
02/18/2003 16:00 To 02/24/2003 08:00 136.00 Hours, Delivered Beam: 133.70 Hours,
02/26/2003 08:00 To 03/03/2003 16:00 128.00 Hours, Delivered Beam: 124.72 Hours,
03/04/2003 08:00 To 03/06/2003 09:04 49.07 Hours, Delivered Beam: 49.07 Hours,

4 Fault(s),4 Fault(s),4 Fault(s),4 Fault(s),9 Fault(s),0 Fault(s),30.59 MTBF, 95.59% of Sched. Time35.42 MTBF, 98.40% of Sched. Time35.08 MTBF, 97.45% of Sched. Time
33.42 MTBF, 98.31% of Sched. Time13.86 MTBF, 97.44% of Sched. Time49.07 MTBF, 100.01% of Sched. Time

Delivered Beam
711.87 Hours
Percentage of Scheduled Time (*)
97.64 %
Downtime During Period
17.17 Hours
Percentage of scheduled time SR current > 10 ma 98.30 %Average Delivered Current During This Period
98.51 ma
Total integrated Current During This Period
70.13 A-hr
Mean Fill Duration in Period
Mean Fill Duration from Poisson Fit
Mean Time Between Faults (MTBF)
Faults per Day of Delivered Beam

27.38 Hours
31.46 Hours
28.47 Hours
0.84

LengthDowntime is associated with the end of a fill.
Valid fills Beginning in this Time Interval|
Reason for|of
The first fill of a period will have any
Fill#
Start
End
Duration |Fill Termination| Downtime
downtime before the fill on the line above.
-----------------------------------------------------------------------------------------------------------------------------------|| 0.00# 1 01/29 08:00 To 01/30 07:44
23.75 | Correct P0 Err.
[CTL] | 0.85 Recover, refill# 3 01/30 08:36 To 01/30 13:59
5.38 | line power bump [OTHER] | 2.08 Investigation, Recovery of injector[OAG], refill# 4 01/30 16:03 To 02/02 02:28
58.41 | 18ID Rad. Mon. Trip [HP] | 2.52 Repair, RF rad monitor problems# 10 02/02 04:59 To 02/02 05:45
0.77 | 22ID BPLD trip
[UKN] | 0.18 refill# 11 02/02 05:56 To 02/03 15:59
34.05 | Int Dump: End of Period | 0.00-----------------------------------------------------------------------------------------------------------------------------------|| 0.19 Lost beam when engaging RTFB system [ES]# 15 02/04 08:11 To 02/07 08:34
72.38 | rf4 Vac trip
[RF] | 0.73 Pump down, refill# 16 02/07 09:18 To 02/07 16:19
7.02 | rf4 llrf trip
[RF] | 0.37 Refill# 17 02/07 16:41 To 02/07 17:24
0.72 | Beam instability[RF] | 0.72 Booster problem [ES], difficulty injecting[PHY]# 18 02/07 18:08 To 02/09 13:36
43.48 | rf4 Vac trip
[RF] | 0.29 Refill# 20 02/09 13:54 To 02/10 07:59
18.09 | Int Dump: End of Period | 0.00

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY YaoCONCLUSION• The APS storage ring operation performance has been
consistent and continuously improving• The organization, training, operational procedures, andmethods implemented by the Operations Group are
successful and effective.

03/10/2003 WAO 2003,
Hayama/Tsukuba,
Japan

Operations Group
Advanced Photon Source
CY Yao