HOW WE DO BUSINESS AT APS *C.Y. Yao, Greg Banks, Glenn Decker, Roger Klaffky
ANL, Argonne, IL 60439, USA
Abstract
The Advanced Photon Source (APS) storage ring is athird-generation dedicated X-ray synchrotron radiation
user facility, which has been in operation since 1996. We
report the organization of the operations team, machineperformance, operation modes, user beam schedule, userinteraction, and other aspects of APS.

1 THE APS FACILITY
The APS storage ring is a 7-GeV electron storage ring.
The present operating current is 100 mA. Its injectorconsists of a 650-MeV electron linear accelerator, a 450MeV electron accumulator ring (PAR) and a 7-GeV
booster synchrotron. Commissioning of the storage ring
started in 1995 and regular operation began in 1996.
Performance of the storage ring has been improving
continuously over the past years. For the first run periodof FY2003, APS reached a record high availability of
99.06% and mean time between faults (MTBF) of 71.76
hours. Figures 1 to 3 show the operational statistics fromFY98 through the first run of FY03.
100.0%

2 THE OPERATIONS GROUP
As a U.S. Department of Energy (DOE) facility, theorganization and standards of APS operations mustcomply with the requirements and guidelines of DOE
Order 5480.19, “Conduct of Operations Requirements forDOE Facilities.”
The APS has three divisions: the Accelerator
Operations Division (AOD), the Accelerator Systems
Division (ASD), and the Experimental Facilities Division(XFD).

80.0

70.0

60.0

50.0

40.0

99.0%
30.0
98.0%
20.0
97.0%
10.0

96.0%

95.0%

0.0
FY1998FY1999FY2000FY2001

FY2002

FY2003

94.0%

93.0%

Fig. 2 Mean time between faults.

92.0%

91.0%

90.0%
FY1998FY1999FY2000FY2001

FY2002

FY2003Fig. 1 User beam availability.

___________________________________________*Work supported by U.S. Department of Energy, Office of Basic
Energy Sciences, under Contract No. W-31-109-ENG-38.

The AOD is responsible for the operation of theaccelerator facilities, experimental area, and site-wide
infrastructure. The ASD is responsible for maintenance,
upgrading, development of accelerator subsystems, aswell as research in the accelerator field. The XFD is
responsible for insertion device and beamline
instrumentation support, new device development, and Xray research.
The Operations Group (OPS) is a part of AOD,
responsible for daily operation of the APS storage ringand its injectors. The OPS Group interacts directly withother groups, in particular the Experimental Floor
Operations Group (EFO), Operations Analysis Group,Beam Diagnostics Group, RF Group, Electrical Systems
Group, and Controls Group.

3 THE MAIN CONTROL ROOM
The MCR is the center of many machine-relatedactivities. The size of the MCR is about 2000 ft2, withthree control consoles forming a circle at the center ofMCR. Two of the consoles are dedicated to operations.
One of them serves as a comfort display and work area
for machine studies. There are 11 double-head Sun
workstations, several video monitors, and a few
oscilloscopes.
The control system is a distributed control system
consisting of Sun workstations, VME-based IOCs andEPICS software.

5100.05000.04900.04800.04700.04600.04500.04400.0

4 OPERATOR TRAINING

4300.04200.04100.0
FY1998FY1999FY2000FY2001

FY2002

FY2003Fig. 3 Scheduled user hours.
The OPS Group currently has 22 members, including
the group leader, the group secretary, five staff members,five crew chiefs, and 11 operators. All the accelerators areoperated from one central location --- the main control
room (MCR).
The operators are divided into five crews. Crews workon a 3-shift rotation with one backup week and one rest
week. On weekends, the shifts change to 12 hours toallow operators more time with their families. The backup
week is intended for operators to work on their training,interact with staff members, work on projects, andparticipate in other daily activities. On 40 hour per weekbase, our operators spend 84% of their time on shift work.
The on-shift crew is responsible for machine operation,
shift event logging, fault reporting, and tunnel safety
handling. The technical systems groups provide on-callsupport and perform repairs. During a user run, the crews
are required to perform certain off-hour repair tasks, suchas magnet power supply swapping, rf station switching,etc.
Our operator turnover rate fluctuates from year to yearwith an average of 15% in recent years. The turnover rateis affected by many factors such as rotating shiftschedules, lack of career advancement opportunities, andgeneral employment conditions. The overall mean andmedian years of experience are 2.7 years and 4.1 years,respectively. The crew chiefs have an average of 7.8 yearsof experience. New hires typically have some college
education and several years of industrial experience.
Maintaining a well focused, consistently performing,and motivated operations team is probably the biggest
challenge for the OPS Group.

The training required for operator qualification isdivided into three categories: the Argonne-specified
training program, general operations training, andaccelerator-specific training. Argonne and APS provide
training courses for the first two categories, such as tunnel
safety, electrical safety, emergency response, radiationsafety, computer security, etc. The OPS Group provides
accelerator-specific training.
A new operator is required to complete qualification on
all four accelerators within two years. On-shift training
and self-guided training are the primary methods.
Training documents and courses are developed byresponsible operations staff and technical systems groups.
Videotapes of the training courses are available forviewing. Crew chiefs and qualified operators conduct onshift training. Qualification tests are administered by OPS
staff members. Re-qualification is required every twoyears after initial qualification. Average qualification timeis 12.2 months from date of hire.

5 OPERATIONS LOGBOOKThe operations logbook exists in both electronic andpaper form. The crews enter logbook entries using
FrameMaker software. A hardcopy is printed at the end ofeach shift. Online web pages are automatically generated
from the FrameMaker file and updated periodically.
The electronic logbook mainly serves as acommunication tool. The hardcopies are kept for archival
purposes. A search tool is used to review and reconstructpast events.

6 OPERATION MODES
Various storage ring operation modes are developed toserve various user requirements. The following are thecurrent operation modes:

•
•
•
•

Top-up, low emittance, 1+1x22 singlets fillpattern
Top-up, low emittance, 1+ 7x8 hybrids fillpatternNon-top-up, high emittance, 1+1x22 singlets
fill pattern
Non-top-up, low emittance, 324 bunch fillpattern

Top-up injection mode provides higher average photon
flux and better stability of both storage ring and beamline.
Lower emittance and higher photon brightness can also beachieved without beam lifetime concern. Top-up mode isscheduled for 75% of the time.
Non-top-up mode is mainly scheduled to provide
injector beam time for parasitic injector study, operator
training, and injector maintenance and improvement.
High emittance lattice or multiple-bunch fill patterns areused in order to increase beam lifetime. Refill isperformed twice a day.
The bunch fill pattern typically has one BPM bunch for
BPM timing, and either singlets or multiple bunch trains
for beamline users.
New lattices and fill patterns are still being developed
to accommodate increasingly diverse user demands.

7 USER INTERACTION
The on-shift crew provides machine status information
with the users through web pages, logbook entries, andPA announcements. User requests, such as beam orbit
steering, are passed on to the crew via the EFO Group. If
necessary, a direct contact between OPS staff members
and beamline users is established to resolve a request.

8 MAINTENANCE PROGRAM
Given the budget constraints and the unpredictability ofmany accelerator components, it is hard to form a uniformmaintenance strategy. Our approach is a combination of“run it until it breaks” and preventive. Many types ofequipment, such as storage ring rf stations, magnet power
supplies, and control system equipment, have swappable
spare units. Offline maintenance can be performed.
Preventive maintenance is performed on storage ring
dipole, kicker, and septum supplies, which do not havespares. Periodic preventive maintenance is performed onmechanical and electrical equipment, where maintenancestandards are available.
We are considering development of trend-analysisbased failure prediction in order to facilitate preventive
maintenance.
The OPS Group’s responsibilities are mainly in theareas of work approval, scheduling, coordination, andrecord keeping. The systems groups submit maintenancerequests, perform work, and verify the functioning ofequipment.

9 SOFTWARE TOOLS
Many software tools have been developed to assist theoperators, which have contributed to better efficiency,productivity, and consistency. These include:
Work request system
RM&D system (repair, maintenance anddevelopment)
Downtime report tool
Operation statistics tool
OPS Group mail list server/archive
Automatic work station configuration tool
Save/restore/compare tool for machine
configuration
Procedure Execution Manager
Scope waveform archiving andconfiguration tool
Data monitoring and review

10 DOWNTIME STATISTICS
Downtime statistics are part of the performancemonitoring and improvement mechanism. The APS
management has set an overall goal of 95% availability
and an MTBF of 48 hours. Every group is also assigned adowntime allowance and an MTBF goal.
Storage ring faults and downtimes are automaticallylogged. The OPS Group performs downtime analyses andposts downtime statistics on a weekly basis. Disputed
items are discussed, resolved, and corrected.

11 CONCLUSION
The APS storage ring operation performance isconsistent and continuously improving. The organization,training,operational
procedures,
andmethods
implemented by the Operations Group are successful andeffective.